* [PATCH RFC v6 1/5] mm/migrate: skip data copy for already-copied folios
2026-06-30 7:28 [PATCH RFC v6 0/5] Accelerate page migration with batch copying and hardware offload Shivank Garg
@ 2026-06-30 7:28 ` Shivank Garg
2026-06-30 7:28 ` [PATCH RFC v6 2/5] mm/migrate: add batch-copy path in migrate_pages_batch Shivank Garg
` (4 subsequent siblings)
5 siblings, 0 replies; 12+ messages in thread
From: Shivank Garg @ 2026-06-30 7:28 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Zi Yan, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price, Ying Huang,
Alistair Popple, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko
Cc: Karim Manaouil, Frank van der Linden, Teja Vojjala,
Pravin Tamkhane, Kinsey Ho, Wei Xu, Matthew Wilcox,
Davidlohr Bueso, Vinod Koul, Bharata B Rao, SeongJae Park,
David Rientjes, Xuezheng Chu, Yiannis Nikolakopoulos, Dave Hansen,
Johannes Weiner, John Hubbard, Peter Xu, Rik van Riel,
Shakeel Butt, Tejun Heo, Fan Ni, Jonathan Cameron,
Aneesh Kumar K.V, Nathan Lynch, Frank Li, Dan Williams, linux-mm,
linux-kernel, Shivank Garg
Add FOLIO_CONTENT_COPIED marker to the dst->migrate_info migration
state. When set, it tells __migrate_folio() to skip folio_mc_copy()
and perform metadata-only migration. This bit is not set yet, the
batch-copy path enables it later in a subsequent patch.
Update __migrate_folio_extract() to preserve FOLIO_CONTENT_COPIED
while it consumes old states of src folio.
Move the dst->migrate_info state enum to migrate.h header file so
offload drivers can see FOLIO_CONTENT_COPIED.
On 32Bit, we do not have spare bit to store FOLIO_CONTENT_COPIED
info, migration copy offload is disabled for them.
Signed-off-by: Shivank Garg <shivankg@amd.com>
---
include/linux/migrate.h | 25 +++++++++++++++++++++++++
mm/migrate.c | 29 +++++++++++++----------------
2 files changed, 38 insertions(+), 16 deletions(-)
diff --git a/include/linux/migrate.h b/include/linux/migrate.h
index d5af2b7f577b..876035df2fdc 100644
--- a/include/linux/migrate.h
+++ b/include/linux/migrate.h
@@ -72,6 +72,31 @@ int folio_migrate_mapping(struct address_space *mapping,
struct folio *newfolio, struct folio *folio, int extra_count);
int set_movable_ops(const struct movable_operations *ops, enum pagetype type);
+/*
+ * To record some information during migration, we use the migrate_info
+ * field of struct folio of the newly allocated destination folio,
+ * together with the anon_vma pointer. The state is encoded in the
+ * unused low bits of the pointer.
+ * This is safe because nobody is using it except us.
+ *
+ * FOLIO_WAS_MAPPED and FOLIO_WAS_MLOCKED record the src folios's state
+ * at unmap time and are consumed during the move/undo phase.
+ * FOLIO_CONTENT_COPIED tells __migrate_folio() that the folio contents
+ * have already been copied, so the per-folio copy can be skipped. A
+ * driver must set this bit on each dst folio it copied.
+ */
+enum {
+ FOLIO_WAS_MAPPED = BIT(0),
+ FOLIO_WAS_MLOCKED = BIT(1),
+ FOLIO_OLD_STATES = FOLIO_WAS_MAPPED | FOLIO_WAS_MLOCKED,
+#ifdef CONFIG_64BIT
+ FOLIO_CONTENT_COPIED = BIT(2),
+#else
+ /* On 32bit we do not have a spare bit, migration-copy offload is disabled. */
+ FOLIO_CONTENT_COPIED = 0,
+#endif
+};
+
#else
static inline void putback_movable_pages(struct list_head *l) {}
diff --git a/mm/migrate.c b/mm/migrate.c
index 7a83032a14b5..b3f632575c82 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -859,14 +859,21 @@ static int __migrate_folio(struct address_space *mapping, struct folio *dst,
enum migrate_mode mode)
{
int rc, expected_count = folio_expected_ref_count(src) + 1;
+ const bool already_copied = dst->migrate_info & FOLIO_CONTENT_COPIED;
+
+ /* Consume the content copied marker */
+ if (already_copied)
+ dst->migrate_info &= ~FOLIO_CONTENT_COPIED;
/* Check whether src does not have extra refs before we do more work */
if (folio_ref_count(src) != expected_count)
return -EAGAIN;
- rc = folio_mc_copy(dst, src);
- if (unlikely(rc))
- return rc;
+ if (!already_copied) {
+ rc = folio_mc_copy(dst, src);
+ if (unlikely(rc))
+ return rc;
+ }
rc = __folio_migrate_mapping(mapping, dst, src, expected_count);
if (rc)
@@ -1129,17 +1136,6 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,
return rc;
}
-/*
- * To record some information during migration, we use the migrate_info
- * field of struct folio of the newly allocated destination folio.
- * This is safe because nobody is using it except us.
- */
-enum {
- FOLIO_WAS_MAPPED = BIT(0),
- FOLIO_WAS_MLOCKED = BIT(1),
- FOLIO_OLD_STATES = FOLIO_WAS_MAPPED | FOLIO_WAS_MLOCKED,
-};
-
static void __migrate_folio_record(struct folio *dst,
int old_folio_state, struct anon_vma *anon_vma)
{
@@ -1151,9 +1147,10 @@ static void __migrate_folio_extract(struct folio *dst,
{
unsigned long info = dst->migrate_info;
- *anon_vmap = (struct anon_vma *)(info & ~FOLIO_OLD_STATES);
+ *anon_vmap = (struct anon_vma *)(info & ~(FOLIO_OLD_STATES |
+ FOLIO_CONTENT_COPIED));
*old_folio_state = info & FOLIO_OLD_STATES;
- dst->migrate_info = 0;
+ dst->migrate_info &= FOLIO_CONTENT_COPIED;
}
/* Restore the source folio to the original state upon failure */
--
2.43.0
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH RFC v6 2/5] mm/migrate: add batch-copy path in migrate_pages_batch
2026-06-30 7:28 [PATCH RFC v6 0/5] Accelerate page migration with batch copying and hardware offload Shivank Garg
2026-06-30 7:28 ` [PATCH RFC v6 1/5] mm/migrate: skip data copy for already-copied folios Shivank Garg
@ 2026-06-30 7:28 ` Shivank Garg
2026-06-30 7:28 ` [PATCH RFC v6 3/5] mm/migrate: add copy offload registration infrastructure Shivank Garg
` (3 subsequent siblings)
5 siblings, 0 replies; 12+ messages in thread
From: Shivank Garg @ 2026-06-30 7:28 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Zi Yan, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price, Ying Huang,
Alistair Popple, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko
Cc: Karim Manaouil, Frank van der Linden, Teja Vojjala,
Pravin Tamkhane, Kinsey Ho, Wei Xu, Matthew Wilcox,
Davidlohr Bueso, Vinod Koul, Bharata B Rao, SeongJae Park,
David Rientjes, Xuezheng Chu, Yiannis Nikolakopoulos, Dave Hansen,
Johannes Weiner, John Hubbard, Peter Xu, Rik van Riel,
Shakeel Butt, Tejun Heo, Fan Ni, Jonathan Cameron,
Aneesh Kumar K.V, Nathan Lynch, Frank Li, Dan Williams, linux-mm,
linux-kernel, Shivank Garg
Add migrate_folios_mc_copy() which walks src/dst folio lists in
lockstep, copies folio content via folio_mc_copy() and marks the dst
with FOLIO_CONTENT_COPIED so the move phase in __migrate_folio() can
skip the per-folio copy. The folios_cnt parameter is unused here,
present for the offload_copy callback signature compatibility used by
later patches in the series.
Split unmapped folios into batch-eligible (unmap_batch/dst_batch) and
standard (unmap_single/dst_single) lists. A folio is batch-eligible
when the migration reason is allowed by the driver, refcount equals
expected and it's not a movable_ops page.
After TLB flush, batch copy the eligible folios via
migrate_folios_mc_copy(), then run the move phase. The
FOLIO_CONTENT_COPIED marker is consumed on first read in the
__migrate_folio, so a -EAGAIN retry falls back to fresh per-folio copy.
Signed-off-by: Shivank Garg <shivankg@amd.com>
---
include/linux/migrate.h | 3 ++
mm/migrate.c | 107 ++++++++++++++++++++++++++++++++++++++++++------
2 files changed, 97 insertions(+), 13 deletions(-)
diff --git a/include/linux/migrate.h b/include/linux/migrate.h
index 876035df2fdc..e62c974b81d5 100644
--- a/include/linux/migrate.h
+++ b/include/linux/migrate.h
@@ -71,6 +71,9 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio);
int folio_migrate_mapping(struct address_space *mapping,
struct folio *newfolio, struct folio *folio, int extra_count);
int set_movable_ops(const struct movable_operations *ops, enum pagetype type);
+int migrate_folios_mc_copy(struct list_head *dst_list,
+ struct list_head *src_list,
+ unsigned int __always_unused folios_cnt);
/*
* To record some information during migration, we use the migrate_info
diff --git a/mm/migrate.c b/mm/migrate.c
index b3f632575c82..41b732e78a67 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -51,6 +51,12 @@
#include "internal.h"
#include "swap.h"
+/* For now, never offload. Wired up in later patch. */
+static bool migrate_should_offload(int reason)
+{
+ return false;
+}
+
static const struct movable_operations *offline_movable_ops;
static const struct movable_operations *zsmalloc_movable_ops;
@@ -1707,6 +1713,60 @@ static int migrate_hugetlbs(struct list_head *from, new_folio_t get_new_folio,
return nr_failed;
}
+static bool folio_can_batch_copy(struct folio *folio)
+{
+ /* movable_ops pages have a separate migration path */
+ if (unlikely(page_has_movable_ops(&folio->page)))
+ return false;
+
+ /*
+ * Only batch when refcount matches the expected refcount.
+ * Otherwise a GUP holder could modify src between the
+ * batch copy and the move-phase refcount check, leaving the
+ * dst folio with a stale copy.
+ */
+ return folio_ref_count(folio) == folio_expected_ref_count(folio) + 1;
+}
+
+/**
+ * migrate_folios_mc_copy - Copy the contents of a list of folios.
+ * @dst_list: destination folio list.
+ * @src_list: source folio list.
+ * @folios_cnt: unused here, present for callback signature compatibility.
+ *
+ * Walk src and dst folio lists in lockstep, copy each folio via
+ * folio_mc_copy(), and mark the dst with FOLIO_CONTENT_COPIED so the
+ * move phase in __migrate_folio() skips the per-folio copy. The caller
+ * must ensure both lists have the same number of entries.
+ *
+ * On error, dst folios already copied keep their FOLIO_CONTENT_COPIED
+ * marker, while the remaining folios fallback to per-folio CPU copy in
+ * the move phase.
+ *
+ * This function may sleep.
+ *
+ * Return: 0 on success, negative errno on failure.
+ */
+int migrate_folios_mc_copy(struct list_head *dst_list,
+ struct list_head *src_list,
+ unsigned int __always_unused folios_cnt)
+{
+ struct folio *src, *dst;
+ int ret;
+
+ dst = list_first_entry(dst_list, struct folio, lru);
+ list_for_each_entry(src, src_list, lru) {
+ ret = folio_mc_copy(dst, src);
+ if (ret)
+ return ret;
+ dst->migrate_info |= FOLIO_CONTENT_COPIED;
+ dst = list_next_entry(dst, lru);
+ cond_resched();
+ }
+
+ return 0;
+}
+
static void migrate_folios_move(struct list_head *src_folios,
struct list_head *dst_folios,
free_folio_t put_new_folio, unsigned long private,
@@ -1806,9 +1866,13 @@ static int migrate_pages_batch(struct list_head *from,
bool is_large = false;
struct folio *folio, *folio2, *dst = NULL;
int rc, rc_saved = 0, nr_pages;
- LIST_HEAD(unmap_folios);
- LIST_HEAD(dst_folios);
+ unsigned int nr_batch = 0;
+ LIST_HEAD(unmap_batch);
+ LIST_HEAD(dst_batch);
+ LIST_HEAD(unmap_single);
+ LIST_HEAD(dst_single);
bool nosplit = (reason == MR_NUMA_MISPLACED);
+ bool offload = migrate_should_offload(reason);
VM_WARN_ON_ONCE(mode != MIGRATE_ASYNC &&
!list_empty(from) && !list_is_singular(from));
@@ -1902,8 +1966,8 @@ static int migrate_pages_batch(struct list_head *from,
private, folio, &dst, mode, ret_folios);
/*
* The rules are:
- * 0: folio will be put on unmap_folios list,
- * dst folio put on dst_folios list
+ * 0: folio put on unmap_batch or unmap_single,
+ * dst folio put on dst_batch or dst_single
* -EAGAIN: stay on the from list
* -ENOMEM: stay on the from list
* Other errno: put on ret_folios list
@@ -1944,7 +2008,7 @@ static int migrate_pages_batch(struct list_head *from,
/* nr_failed isn't updated for not used */
stats->nr_thp_failed += thp_retry;
rc_saved = rc;
- if (list_empty(&unmap_folios))
+ if (list_empty(&unmap_batch) && list_empty(&unmap_single))
goto out;
else
goto move;
@@ -1954,8 +2018,14 @@ static int migrate_pages_batch(struct list_head *from,
nr_retry_pages += nr_pages;
break;
case 0:
- list_move_tail(&folio->lru, &unmap_folios);
- list_add_tail(&dst->lru, &dst_folios);
+ if (offload && folio_can_batch_copy(folio)) {
+ list_move_tail(&folio->lru, &unmap_batch);
+ list_add_tail(&dst->lru, &dst_batch);
+ nr_batch++;
+ } else {
+ list_move_tail(&folio->lru, &unmap_single);
+ list_add_tail(&dst->lru, &dst_single);
+ }
break;
default:
/*
@@ -1978,17 +2048,26 @@ static int migrate_pages_batch(struct list_head *from,
/* Flush TLBs for all unmapped folios */
try_to_unmap_flush();
+ /* Batch-copy eligible folios before the move phase */
+ if (!list_empty(&unmap_batch))
+ migrate_folios_mc_copy(&dst_batch, &unmap_batch, nr_batch);
+
retry = 1;
for (pass = 0; pass < nr_pass && retry; pass++) {
retry = 0;
thp_retry = 0;
nr_retry_pages = 0;
- /* Move the unmapped folios */
- migrate_folios_move(&unmap_folios, &dst_folios,
- put_new_folio, private, mode, reason,
- ret_folios, stats, &retry, &thp_retry,
- &nr_failed, &nr_retry_pages);
+ if (!list_empty(&unmap_batch))
+ migrate_folios_move(&unmap_batch, &dst_batch, put_new_folio,
+ private, mode, reason, ret_folios, stats,
+ &retry, &thp_retry, &nr_failed,
+ &nr_retry_pages);
+ if (!list_empty(&unmap_single))
+ migrate_folios_move(&unmap_single, &dst_single, put_new_folio,
+ private, mode, reason, ret_folios, stats,
+ &retry, &thp_retry, &nr_failed,
+ &nr_retry_pages);
}
nr_failed += retry;
stats->nr_thp_failed += thp_retry;
@@ -1997,7 +2076,9 @@ static int migrate_pages_batch(struct list_head *from,
rc = rc_saved ? : nr_failed;
out:
/* Cleanup remaining folios */
- migrate_folios_undo(&unmap_folios, &dst_folios,
+ migrate_folios_undo(&unmap_batch, &dst_batch,
+ put_new_folio, private, ret_folios);
+ migrate_folios_undo(&unmap_single, &dst_single,
put_new_folio, private, ret_folios);
return rc;
--
2.43.0
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH RFC v6 3/5] mm/migrate: add copy offload registration infrastructure
2026-06-30 7:28 [PATCH RFC v6 0/5] Accelerate page migration with batch copying and hardware offload Shivank Garg
2026-06-30 7:28 ` [PATCH RFC v6 1/5] mm/migrate: skip data copy for already-copied folios Shivank Garg
2026-06-30 7:28 ` [PATCH RFC v6 2/5] mm/migrate: add batch-copy path in migrate_pages_batch Shivank Garg
@ 2026-06-30 7:28 ` Shivank Garg
2026-07-20 11:19 ` Huang, Ying
2026-06-30 7:28 ` [PATCH RFC v6 4/5] drivers/migrate_offload: add DMA batch copy driver (dcbm) Shivank Garg
` (2 subsequent siblings)
5 siblings, 1 reply; 12+ messages in thread
From: Shivank Garg @ 2026-06-30 7:28 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Zi Yan, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price, Ying Huang,
Alistair Popple, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko
Cc: Karim Manaouil, Frank van der Linden, Teja Vojjala,
Pravin Tamkhane, Kinsey Ho, Wei Xu, Matthew Wilcox,
Davidlohr Bueso, Vinod Koul, Bharata B Rao, SeongJae Park,
David Rientjes, Xuezheng Chu, Yiannis Nikolakopoulos, Dave Hansen,
Johannes Weiner, John Hubbard, Peter Xu, Rik van Riel,
Shakeel Butt, Tejun Heo, Fan Ni, Jonathan Cameron,
Aneesh Kumar K.V, Nathan Lynch, Frank Li, Dan Williams, linux-mm,
linux-kernel, Shivank Garg, Mike Day
Add a registration interface that lets a single offload provider
(DMA, multi-threaded CPU copy, etc) take over the batch folio copy
performed by migrate_pages_batch().
The provider fills in a struct migrator with an offload_copy()
callback and calls migrate_offload_register(), passing the set of
migration reasons it wants to handle. Registration updates the
migrate_offload_batch_copy_fn() static_call and enables the
migrate_offload_enabled static branch. migrate_offload_unregister()
reverts both.
The active_reason_mask is the single source of truth for which migration
reasons are eligible for batched copy and offload. This is checked with
the migrate_should_offload(). The active reason mask can be changed at
runtime via migrate_offload_set_reason_mask(). Also, add
migrate_offload_reason_mask_parse() and
migrate_offload_reason_mask_format() so a provider can expose its reason
mask using migrate_reason_names[] (e.g. "compaction,demotion", "all",
"none").
Only one migrator can be active at a time. A second registration
gets -EBUSY, and only the active migrator can unregister itself.
The static_call dispatch is protected by SRCU so that the
synchronize_srcu() in unregister waits for all in-flight copy before
the module reference is dropped.
Callbacks must set FOLIO_CONTENT_COPIED on each successfully copied
dst folio so __migrate_folio() skips the per-folio copy; folios
without the marker fall back to CPU copy in the move phase.
Co-developed-by: Mike Day <michael.day@amd.com>
Signed-off-by: Mike Day <michael.day@amd.com>
Signed-off-by: Shivank Garg <shivankg@amd.com>
---
MAINTAINERS | 2 +
include/linux/migrate_copy_offload.h | 69 ++++++++++
mm/Kconfig | 6 +
mm/Makefile | 1 +
mm/migrate.c | 9 +-
mm/migrate_copy_offload.c | 249 +++++++++++++++++++++++++++++++++++
6 files changed, 329 insertions(+), 7 deletions(-)
diff --git a/MAINTAINERS b/MAINTAINERS
index 762355b62d54..fa32b447f3e5 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -17065,9 +17065,11 @@ F: Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-weighted-interleave
F: include/linux/mempolicy.h
F: include/uapi/linux/mempolicy.h
F: include/linux/migrate.h
+F: include/linux/migrate_copy_offload.h
F: include/linux/migrate_mode.h
F: mm/mempolicy.c
F: mm/migrate.c
+F: mm/migrate_copy_offload.c
F: mm/migrate_device.c
MEMORY MANAGEMENT - MGLRU (MULTI-GEN LRU)
diff --git a/include/linux/migrate_copy_offload.h b/include/linux/migrate_copy_offload.h
new file mode 100644
index 000000000000..97941ebd4979
--- /dev/null
+++ b/include/linux/migrate_copy_offload.h
@@ -0,0 +1,69 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_MIGRATE_COPY_OFFLOAD_H
+#define _LINUX_MIGRATE_COPY_OFFLOAD_H
+
+#include <linux/bits.h>
+#include <linux/errno.h>
+#include <linux/migrate_mode.h>
+
+struct list_head;
+struct module;
+
+#define MIGRATOR_NAME_LEN 32
+
+/*
+ * Reasons that may ever be offloaded. A driver's mask is clamped to this.
+ * Allow all reasons by default.
+ */
+#define MIGRATE_OFFLOAD_REASONS_ALLOWED (BIT(MR_TYPES) - 1)
+
+/**
+ * struct migrator - batch-copy provider for page migration.
+ * @name: name of the provider.
+ * @offload_copy: copy @folio_cnt folios from @src_list to @dst_list.
+ *
+ * The migrator may inspect @folio_cnt to decide whether the batch
+ * is worth offloading, e.g. skip when the batch is too small to
+ * amortize setup cost.
+ * The callback must set FOLIO_CONTENT_COPIED in dst->migrate_info
+ * for each successfully copied destination folio.
+ * Folios without this marker are copied via per-folio CPU copy in
+ * the move phase.
+ *
+ * @owner: module providing the migrator. NULL for built-in (=y) drivers.
+ */
+struct migrator {
+ char name[MIGRATOR_NAME_LEN];
+ int (*offload_copy)(struct list_head *dst_list, struct list_head *src_list,
+ unsigned int folio_cnt);
+ struct module *owner;
+};
+
+#ifdef CONFIG_MIGRATION_COPY_OFFLOAD
+int migrate_offload_register(const struct migrator *m, unsigned long reason_mask);
+int migrate_offload_unregister(const struct migrator *m);
+int migrate_offload_set_reason_mask(const struct migrator *m, unsigned long mask);
+int migrate_offload_reason_mask_parse(const char *buf, unsigned long *maskp);
+int migrate_offload_reason_mask_format(char *buf, unsigned long mask);
+bool migrate_should_offload(int reason);
+int migrate_offload_batch_copy(struct list_head *dst_batch,
+ struct list_head *src_batch, unsigned int nr_batch);
+#else
+static inline int migrate_offload_register(const struct migrator *m,
+ unsigned long reason_mask) { return -EOPNOTSUPP; }
+static inline int migrate_offload_unregister(const struct migrator *m) { return -EOPNOTSUPP; }
+static inline int migrate_offload_set_reason_mask(const struct migrator *m,
+ unsigned long mask) { return -EOPNOTSUPP; }
+static inline int migrate_offload_reason_mask_parse(const char *buf,
+ unsigned long *maskp) { return -EOPNOTSUPP; }
+static inline int migrate_offload_reason_mask_format(char *buf,
+ unsigned long mask) { return -EOPNOTSUPP; }
+static inline bool migrate_should_offload(int reason) { return false; }
+static inline int migrate_offload_batch_copy(struct list_head *dst_batch,
+ struct list_head *src_batch, unsigned int nr_batch)
+{
+ return -EOPNOTSUPP;
+}
+#endif
+
+#endif /* _LINUX_MIGRATE_COPY_OFFLOAD_H */
diff --git a/mm/Kconfig b/mm/Kconfig
index 9e0ca4824905..c65639bae268 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -700,6 +700,12 @@ config MIGRATION
config DEVICE_MIGRATION
def_bool MIGRATION && ZONE_DEVICE
+# Page-migration batch-copy offload infrastructure.
+# Selected by migrator drivers (e.g. CONFIG_DCBM_DMA).
+config MIGRATION_COPY_OFFLOAD
+ bool
+ depends on MIGRATION && 64BIT
+
config ARCH_ENABLE_HUGEPAGE_MIGRATION
bool
diff --git a/mm/Makefile b/mm/Makefile
index eff9f9e7e061..256fb532b672 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -96,6 +96,7 @@ obj-$(CONFIG_FAILSLAB) += failslab.o
obj-$(CONFIG_FAIL_PAGE_ALLOC) += fail_page_alloc.o
obj-$(CONFIG_MEMTEST) += memtest.o
obj-$(CONFIG_MIGRATION) += migrate.o
+obj-$(CONFIG_MIGRATION_COPY_OFFLOAD) += migrate_copy_offload.o
obj-$(CONFIG_NUMA) += memory-tiers.o
obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o
obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o
diff --git a/mm/migrate.c b/mm/migrate.c
index 41b732e78a67..4fed3110ca0a 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -43,6 +43,7 @@
#include <linux/sched/sysctl.h>
#include <linux/memory-tiers.h>
#include <linux/pagewalk.h>
+#include <linux/migrate_copy_offload.h>
#include <asm/tlbflush.h>
@@ -51,12 +52,6 @@
#include "internal.h"
#include "swap.h"
-/* For now, never offload. Wired up in later patch. */
-static bool migrate_should_offload(int reason)
-{
- return false;
-}
-
static const struct movable_operations *offline_movable_ops;
static const struct movable_operations *zsmalloc_movable_ops;
@@ -2050,7 +2045,7 @@ static int migrate_pages_batch(struct list_head *from,
/* Batch-copy eligible folios before the move phase */
if (!list_empty(&unmap_batch))
- migrate_folios_mc_copy(&dst_batch, &unmap_batch, nr_batch);
+ migrate_offload_batch_copy(&dst_batch, &unmap_batch, nr_batch);
retry = 1;
for (pass = 0; pass < nr_pass && retry; pass++) {
diff --git a/mm/migrate_copy_offload.c b/mm/migrate_copy_offload.c
new file mode 100644
index 000000000000..335db8e80556
--- /dev/null
+++ b/mm/migrate_copy_offload.c
@@ -0,0 +1,249 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bitops.h>
+#include <linux/jump_label.h>
+#include <linux/kstrtox.h>
+#include <linux/migrate.h>
+#include <linux/migrate_copy_offload.h>
+#include <linux/mm.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/srcu.h>
+#include <linux/static_call.h>
+#include <linux/string.h>
+#include <linux/sysfs.h>
+
+DEFINE_STATIC_KEY_FALSE(migrate_offload_enabled);
+DEFINE_SRCU(migrate_offload_srcu);
+DEFINE_STATIC_CALL(migrate_offload_batch_copy_fn, migrate_folios_mc_copy);
+
+static DEFINE_MUTEX(migrator_mutex);
+static const struct migrator *active_migrator;
+
+/*
+ * The active migrator's reason mask. This is the single source of truth
+ * for which reasons are offloaded.
+ */
+static unsigned long active_reason_mask;
+
+bool migrate_should_offload(int reason)
+{
+ if (!static_branch_unlikely(&migrate_offload_enabled))
+ return false;
+
+ return READ_ONCE(active_reason_mask) & BIT(reason);
+}
+
+/**
+ * migrate_offload_reason_mask_parse - parse migration reason mask.
+ * @buf: input string, either a number (hex/decimal, e.g. "0x101") or a
+ * comma-separated list of reason names (see migrate_reason_names[]),
+ * e.g. "compaction,demotion". The tokens "all" and "none" select
+ * every or no reason.
+ * @maskp: parsed mask, clamped to MIGRATE_OFFLOAD_REASONS_ALLOWED.
+ *
+ * Return: 0 on success, -EINVAL on an unknown name, -ENOMEM on OOM.
+ */
+int migrate_offload_reason_mask_parse(const char *buf, unsigned long *maskp)
+{
+ unsigned long mask;
+ char *copy, *p, *tok;
+ int ret = 0;
+
+ /* A plain number is treated as a raw mask. */
+ if (!kstrtoul(buf, 0, &mask))
+ goto done;
+
+ copy = kstrdup(buf, GFP_KERNEL);
+ if (!copy)
+ return -ENOMEM;
+ mask = 0;
+ p = strim(copy);
+
+ while ((tok = strsep(&p, ",")) != NULL) {
+ int i;
+
+ tok = strim(tok);
+ if (!*tok)
+ continue;
+ if (!strcmp(tok, "none")) {
+ mask = 0;
+ continue;
+ }
+ if (!strcmp(tok, "all")) {
+ mask = MIGRATE_OFFLOAD_REASONS_ALLOWED;
+ continue;
+ }
+ i = match_string(migrate_reason_names, MR_TYPES, tok);
+ if (i < 0) {
+ ret = -EINVAL;
+ break;
+ }
+ mask |= BIT(i);
+ }
+ kfree(copy);
+ if (ret)
+ return ret;
+done:
+ *maskp = mask & MIGRATE_OFFLOAD_REASONS_ALLOWED;
+ return 0;
+}
+EXPORT_SYMBOL_GPL(migrate_offload_reason_mask_parse);
+
+/**
+ * migrate_offload_reason_mask_format - render a reason mask as names.
+ * @buf: output buffer from kernel_param_ops get.
+ * @mask: reason mask to render.
+ *
+ * Emits a comma-separated list of reason names, or "none".
+ *
+ * Return: number of bytes written to @buf.
+ */
+int migrate_offload_reason_mask_format(char *buf, unsigned long mask)
+{
+ int len = 0, i;
+
+ for_each_set_bit(i, &mask, MR_TYPES)
+ len += sysfs_emit_at(buf, len, "%s%s",
+ len ? "," : "", migrate_reason_names[i]);
+ if (!len)
+ return sysfs_emit(buf, "none\n");
+
+ len += sysfs_emit_at(buf, len, "\n");
+ return len;
+}
+EXPORT_SYMBOL_GPL(migrate_offload_reason_mask_format);
+
+/*
+ * Hand the batch to the registered migrator. The migrator may decline
+ * (typically based on batch size), in which case the move phase falls
+ * back to per-folio CPU copy.
+ */
+int migrate_offload_batch_copy(struct list_head *dst_batch,
+ struct list_head *src_batch, unsigned int nr_batch)
+{
+ int idx, rc;
+
+ idx = srcu_read_lock(&migrate_offload_srcu);
+ rc = static_call(migrate_offload_batch_copy_fn)(dst_batch, src_batch, nr_batch);
+ srcu_read_unlock(&migrate_offload_srcu, idx);
+ return rc;
+}
+
+/**
+ * migrate_offload_set_reason_mask - update the active migrator's reason mask.
+ * @m: migrator (must be the currently active one).
+ * @mask: new reason mask.
+ *
+ * Return: 0 on success, -EINVAL if @m is NULL or not the active migrator.
+ */
+int migrate_offload_set_reason_mask(const struct migrator *m, unsigned long mask)
+{
+ if (!m)
+ return -EINVAL;
+
+ mask &= MIGRATE_OFFLOAD_REASONS_ALLOWED;
+
+ mutex_lock(&migrator_mutex);
+ if (active_migrator != m) {
+ mutex_unlock(&migrator_mutex);
+ return -EINVAL;
+ }
+ WRITE_ONCE(active_reason_mask, mask);
+ mutex_unlock(&migrator_mutex);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(migrate_offload_set_reason_mask);
+
+/**
+ * migrate_offload_register - register a batch-copy provider for page migration.
+ * @m: migrator to install.
+ * @reason_mask: initial set of BIT(MR_*) reasons to offload, can be changed
+ * later via migrate_offload_set_reason_mask().
+ *
+ * Only one provider can be active at a time, returns -EBUSY if another migrator
+ * is already registered.
+ *
+ * Return: 0 on success, negative errno on failure.
+ */
+int migrate_offload_register(const struct migrator *m, unsigned long reason_mask)
+{
+ unsigned long mask;
+ int ret = 0;
+
+ if (!m || !m->offload_copy)
+ return -EINVAL;
+
+ mask = reason_mask & MIGRATE_OFFLOAD_REASONS_ALLOWED;
+
+ mutex_lock(&migrator_mutex);
+ if (active_migrator) {
+ ret = -EBUSY;
+ goto unlock;
+ }
+
+ /* @owner is NULL for built-in (=y) drivers; nothing to ref. */
+ if (m->owner && !try_module_get(m->owner)) {
+ ret = -ENODEV;
+ goto unlock;
+ }
+
+ WRITE_ONCE(active_reason_mask, mask);
+ static_call_update(migrate_offload_batch_copy_fn, m->offload_copy);
+ active_migrator = m;
+ static_branch_enable(&migrate_offload_enabled);
+
+unlock:
+ mutex_unlock(&migrator_mutex);
+
+ if (ret)
+ pr_err("migrate_offload: %s: failed to register (%d)\n", m->name, ret);
+ else
+ pr_info("migrate_offload: enabled by %s (reason_mask=0x%lx)\n",
+ m->name, mask);
+ return ret;
+}
+EXPORT_SYMBOL_GPL(migrate_offload_register);
+
+/**
+ * migrate_offload_unregister - unregister the active batch-copy provider.
+ * @m: migrator to remove (must be the currently active one).
+ *
+ * Reverts static_call targets and waits for SRCU grace period so that
+ * no in-flight migration is still calling the driver functions before
+ * releasing the module.
+ *
+ * Return: 0 on success, negative errno on failure.
+ */
+int migrate_offload_unregister(const struct migrator *m)
+{
+ struct module *owner;
+
+ if (!m)
+ return -EINVAL;
+
+ mutex_lock(&migrator_mutex);
+ if (active_migrator != m) {
+ mutex_unlock(&migrator_mutex);
+ return -EINVAL;
+ }
+
+ /*
+ * Disable the static branch first so new migrate_pages_batch() calls
+ * cannot enter the batch path.
+ */
+ static_branch_disable(&migrate_offload_enabled);
+ WRITE_ONCE(active_reason_mask, 0);
+ static_call_update(migrate_offload_batch_copy_fn, migrate_folios_mc_copy);
+ owner = active_migrator->owner;
+ active_migrator = NULL;
+ mutex_unlock(&migrator_mutex);
+
+ /* Wait for all in-flight callers to finish before module_put(). */
+ synchronize_srcu(&migrate_offload_srcu);
+ if (owner)
+ module_put(owner);
+
+ pr_info("migrate_offload: disabled by %s\n", m->name);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(migrate_offload_unregister);
--
2.43.0
^ permalink raw reply related [flat|nested] 12+ messages in thread* Re: [PATCH RFC v6 3/5] mm/migrate: add copy offload registration infrastructure
2026-06-30 7:28 ` [PATCH RFC v6 3/5] mm/migrate: add copy offload registration infrastructure Shivank Garg
@ 2026-07-20 11:19 ` Huang, Ying
2026-07-20 14:44 ` Zi Yan
0 siblings, 1 reply; 12+ messages in thread
From: Huang, Ying @ 2026-07-20 11:19 UTC (permalink / raw)
To: Shivank Garg
Cc: Andrew Morton, David Hildenbrand, Zi Yan, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price,
Alistair Popple, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko,
Karim Manaouil, Frank van der Linden, Teja Vojjala,
Pravin Tamkhane, Kinsey Ho, Wei Xu, Matthew Wilcox,
Davidlohr Bueso, Vinod Koul, Bharata B Rao, SeongJae Park,
David Rientjes, Xuezheng Chu, Yiannis Nikolakopoulos, Dave Hansen,
Johannes Weiner, John Hubbard, Peter Xu, Rik van Riel,
Shakeel Butt, Tejun Heo, Fan Ni, Jonathan Cameron,
Aneesh Kumar K.V, Nathan Lynch, Frank Li, Dan Williams, linux-mm,
linux-kernel, Mike Day
Shivank Garg <shivankg@amd.com> writes:
[snip]
> diff --git a/mm/migrate.c b/mm/migrate.c
> index 41b732e78a67..4fed3110ca0a 100644
> --- a/mm/migrate.c
> +++ b/mm/migrate.c
> @@ -43,6 +43,7 @@
> #include <linux/sched/sysctl.h>
> #include <linux/memory-tiers.h>
> #include <linux/pagewalk.h>
> +#include <linux/migrate_copy_offload.h>
>
> #include <asm/tlbflush.h>
>
> @@ -51,12 +52,6 @@
> #include "internal.h"
> #include "swap.h"
>
> -/* For now, never offload. Wired up in later patch. */
> -static bool migrate_should_offload(int reason)
> -{
> - return false;
> -}
> -
> static const struct movable_operations *offline_movable_ops;
> static const struct movable_operations *zsmalloc_movable_ops;
>
> @@ -2050,7 +2045,7 @@ static int migrate_pages_batch(struct list_head *from,
>
> /* Batch-copy eligible folios before the move phase */
> if (!list_empty(&unmap_batch))
> - migrate_folios_mc_copy(&dst_batch, &unmap_batch, nr_batch);
> + migrate_offload_batch_copy(&dst_batch, &unmap_batch, nr_batch);
>
> retry = 1;
> for (pass = 0; pass < nr_pass && retry; pass++) {
Found a difference between v5 and v6 here. In v5, migrate_pages_batch()
checks the return value of migrate_offload_batch_copy() and acts
accordingly. While in v6, it does not. Why was this changed?
IIUC, FOLIO_CONTENT_COPIED is now used to indicate a copy failure. Is
this necessary? Is it possible that some (but not all) folios fail to
be copied?
[snip]
---
Best Regards,
Huang, Ying
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [PATCH RFC v6 3/5] mm/migrate: add copy offload registration infrastructure
2026-07-20 11:19 ` Huang, Ying
@ 2026-07-20 14:44 ` Zi Yan
2026-07-20 15:32 ` Garg, Shivank
2026-07-21 11:32 ` Huang, Ying
0 siblings, 2 replies; 12+ messages in thread
From: Zi Yan @ 2026-07-20 14:44 UTC (permalink / raw)
To: Huang, Ying
Cc: Shivank Garg, Andrew Morton, David Hildenbrand, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price,
Alistair Popple, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko,
Karim Manaouil, Frank van der Linden, Teja Vojjala,
Pravin Tamkhane, Kinsey Ho, Wei Xu, Matthew Wilcox,
Davidlohr Bueso, Vinod Koul, Bharata B Rao, SeongJae Park,
David Rientjes, Xuezheng Chu, Yiannis Nikolakopoulos, Dave Hansen,
Johannes Weiner, John Hubbard, Peter Xu, Rik van Riel,
Shakeel Butt, Tejun Heo, Fan Ni, Jonathan Cameron,
Aneesh Kumar K.V, Nathan Lynch, Frank Li, Dan Williams, linux-mm,
linux-kernel, Mike Day
On 20 Jul 2026, at 7:19, Huang, Ying wrote:
> Shivank Garg <shivankg@amd.com> writes:
>
> [snip]
>
>> diff --git a/mm/migrate.c b/mm/migrate.c
>> index 41b732e78a67..4fed3110ca0a 100644
>> --- a/mm/migrate.c
>> +++ b/mm/migrate.c
>> @@ -43,6 +43,7 @@
>> #include <linux/sched/sysctl.h>
>> #include <linux/memory-tiers.h>
>> #include <linux/pagewalk.h>
>> +#include <linux/migrate_copy_offload.h>
>>
>> #include <asm/tlbflush.h>
>>
>> @@ -51,12 +52,6 @@
>> #include "internal.h"
>> #include "swap.h"
>>
>> -/* For now, never offload. Wired up in later patch. */
>> -static bool migrate_should_offload(int reason)
>> -{
>> - return false;
>> -}
>> -
>> static const struct movable_operations *offline_movable_ops;
>> static const struct movable_operations *zsmalloc_movable_ops;
>>
>> @@ -2050,7 +2045,7 @@ static int migrate_pages_batch(struct list_head *from,
>>
>> /* Batch-copy eligible folios before the move phase */
>> if (!list_empty(&unmap_batch))
>> - migrate_folios_mc_copy(&dst_batch, &unmap_batch, nr_batch);
>> + migrate_offload_batch_copy(&dst_batch, &unmap_batch, nr_batch);
>>
>> retry = 1;
>> for (pass = 0; pass < nr_pass && retry; pass++) {
>
> Found a difference between v5 and v6 here. In v5, migrate_pages_batch()
> checks the return value of migrate_offload_batch_copy() and acts
> accordingly. While in v6, it does not. Why was this changed?
>
> IIUC, FOLIO_CONTENT_COPIED is now used to indicate a copy failure. Is
> this necessary? Is it possible that some (but not all) folios fail to
> be copied?
Based on the discussion[1], FOLIO_CONTENT_COPIED is used to indicate both
1. folio is copied via offloading, and 2. MC copy failure. So the code
no longer needs to store the return value of migrate_offload_batch_copy()
and can just check dst->migrate_info & FOLIO_CONTENT_COPIED.
For 32bit, FOLIO_CONTENT_COPIED is always 0, so all folios are always
copied in the serialized path. Hmm, that might be causing double copying,
since migrate_folios_mc_copy() copies batched folios and later
migrate_folios_move() copies them again since
dst->migrate_info & FOLIO_CONTENT_COPIED is always false. Maybe we could
make migrate_folios_mc_copy() return immediately in 32bit, otherwise,
we will need to find somewhere else to store FOLIO_CONTENT_COPIED.
[1] https://lore.kernel.org/all/c2bfdf39-8caa-43bd-b1ad-2285b4cc767a@amd.com/
>
> [snip]
>
> ---
> Best Regards,
> Huang, Ying
Best Regards,
Yan, Zi
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [PATCH RFC v6 3/5] mm/migrate: add copy offload registration infrastructure
2026-07-20 14:44 ` Zi Yan
@ 2026-07-20 15:32 ` Garg, Shivank
2026-07-20 15:45 ` Zi Yan
2026-07-21 11:32 ` Huang, Ying
1 sibling, 1 reply; 12+ messages in thread
From: Garg, Shivank @ 2026-07-20 15:32 UTC (permalink / raw)
To: Zi Yan, Huang, Ying
Cc: Andrew Morton, David Hildenbrand, Matthew Brost, Joshua Hahn,
Rakie Kim, Byungchul Park, Gregory Price, Alistair Popple,
Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Karim Manaouil,
Frank van der Linden, Teja Vojjala, Pravin Tamkhane, Kinsey Ho,
Wei Xu, Matthew Wilcox, Davidlohr Bueso, Vinod Koul,
Bharata B Rao, SeongJae Park, David Rientjes, Xuezheng Chu,
Yiannis Nikolakopoulos, Dave Hansen, Johannes Weiner,
John Hubbard, Peter Xu, Rik van Riel, Shakeel Butt, Tejun Heo,
Fan Ni, Jonathan Cameron, Aneesh Kumar K.V, Nathan Lynch,
Frank Li, Dan Williams, linux-mm, linux-kernel, Mike Day
On 7/20/2026 8:14 PM, Zi Yan wrote:
> On 20 Jul 2026, at 7:19, Huang, Ying wrote:
>
>> Shivank Garg <shivankg@amd.com> writes:
>>
>> [snip]
>>
>>> diff --git a/mm/migrate.c b/mm/migrate.c
>>> index 41b732e78a67..4fed3110ca0a 100644
>>> --- a/mm/migrate.c
>>> +++ b/mm/migrate.c
>>> @@ -43,6 +43,7 @@
>>> #include <linux/sched/sysctl.h>
>>> #include <linux/memory-tiers.h>
>>> #include <linux/pagewalk.h>
>>> +#include <linux/migrate_copy_offload.h>
>>>
>>> #include <asm/tlbflush.h>
>>>
>>> @@ -51,12 +52,6 @@
>>> #include "internal.h"
>>> #include "swap.h"
>>>
>>> -/* For now, never offload. Wired up in later patch. */
>>> -static bool migrate_should_offload(int reason)
>>> -{
>>> - return false;
>>> -}
>>> -
>>> static const struct movable_operations *offline_movable_ops;
>>> static const struct movable_operations *zsmalloc_movable_ops;
>>>
>>> @@ -2050,7 +2045,7 @@ static int migrate_pages_batch(struct list_head *from,
>>>
>>> /* Batch-copy eligible folios before the move phase */
>>> if (!list_empty(&unmap_batch))
>>> - migrate_folios_mc_copy(&dst_batch, &unmap_batch, nr_batch);
>>> + migrate_offload_batch_copy(&dst_batch, &unmap_batch, nr_batch);
>>>
>>> retry = 1;
>>> for (pass = 0; pass < nr_pass && retry; pass++) {
>>
>> Found a difference between v5 and v6 here. In v5, migrate_pages_batch()
>> checks the return value of migrate_offload_batch_copy() and acts
>> accordingly. While in v6, it does not. Why was this changed?
>>
>> IIUC, FOLIO_CONTENT_COPIED is now used to indicate a copy failure. Is
>> this necessary? Is it possible that some (but not all) folios fail to
>> be copied?
>
> Based on the discussion[1], FOLIO_CONTENT_COPIED is used to indicate both
> 1. folio is copied via offloading, and 2. MC copy failure. So the code
> no longer needs to store the return value of migrate_offload_batch_copy()
> and can just check dst->migrate_info & FOLIO_CONTENT_COPIED.
>
> For 32bit, FOLIO_CONTENT_COPIED is always 0, so all folios are always
> copied in the serialized path. Hmm, that might be causing double copying,
> since migrate_folios_mc_copy() copies batched folios and later
> migrate_folios_move() copies them again since
> dst->migrate_info & FOLIO_CONTENT_COPIED is always false. Maybe we could
> make migrate_folios_mc_copy() return immediately in 32bit, otherwise,
> we will need to find somewhere else to store FOLIO_CONTENT_COPIED.
>
> [1] https://lore.kernel.org/all/c2bfdf39-8caa-43bd-b1ad-2285b4cc767a@amd.com/
>
>>
It is not cause double copying.
migrate_should_offload() /and migrate_offload_enabled
gate if folio can be added to unmap_batch/dst_batch.
migrate_offload_enabled branch is enabled from the offload driver like DCBM (which
again depends on 64_BIT) will call migrate_offload_register().
I think I can add a defensive 64_BIT check in migrate_should_offload() to prevent
any potential bugs.
Thanks,
Shivank
>> [snip]
>>
>> ---
>> Best Regards,
>> Huang, Ying
>
>
> Best Regards,
> Yan, Zi
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [PATCH RFC v6 3/5] mm/migrate: add copy offload registration infrastructure
2026-07-20 15:32 ` Garg, Shivank
@ 2026-07-20 15:45 ` Zi Yan
0 siblings, 0 replies; 12+ messages in thread
From: Zi Yan @ 2026-07-20 15:45 UTC (permalink / raw)
To: Garg, Shivank
Cc: Huang, Ying, Andrew Morton, David Hildenbrand, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price,
Alistair Popple, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko,
Karim Manaouil, Frank van der Linden, Teja Vojjala,
Pravin Tamkhane, Kinsey Ho, Wei Xu, Matthew Wilcox,
Davidlohr Bueso, Vinod Koul, Bharata B Rao, SeongJae Park,
David Rientjes, Xuezheng Chu, Yiannis Nikolakopoulos, Dave Hansen,
Johannes Weiner, John Hubbard, Peter Xu, Rik van Riel,
Shakeel Butt, Tejun Heo, Fan Ni, Jonathan Cameron,
Aneesh Kumar K.V, Nathan Lynch, Frank Li, Dan Williams, linux-mm,
linux-kernel, Mike Day
On 20 Jul 2026, at 11:32, Garg, Shivank wrote:
> On 7/20/2026 8:14 PM, Zi Yan wrote:
>> On 20 Jul 2026, at 7:19, Huang, Ying wrote:
>>
>>> Shivank Garg <shivankg@amd.com> writes:
>>>
>>> [snip]
>>>
>>>> diff --git a/mm/migrate.c b/mm/migrate.c
>>>> index 41b732e78a67..4fed3110ca0a 100644
>>>> --- a/mm/migrate.c
>>>> +++ b/mm/migrate.c
>>>> @@ -43,6 +43,7 @@
>>>> #include <linux/sched/sysctl.h>
>>>> #include <linux/memory-tiers.h>
>>>> #include <linux/pagewalk.h>
>>>> +#include <linux/migrate_copy_offload.h>
>>>>
>>>> #include <asm/tlbflush.h>
>>>>
>>>> @@ -51,12 +52,6 @@
>>>> #include "internal.h"
>>>> #include "swap.h"
>>>>
>>>> -/* For now, never offload. Wired up in later patch. */
>>>> -static bool migrate_should_offload(int reason)
>>>> -{
>>>> - return false;
>>>> -}
>>>> -
>>>> static const struct movable_operations *offline_movable_ops;
>>>> static const struct movable_operations *zsmalloc_movable_ops;
>>>>
>>>> @@ -2050,7 +2045,7 @@ static int migrate_pages_batch(struct list_head *from,
>>>>
>>>> /* Batch-copy eligible folios before the move phase */
>>>> if (!list_empty(&unmap_batch))
>>>> - migrate_folios_mc_copy(&dst_batch, &unmap_batch, nr_batch);
>>>> + migrate_offload_batch_copy(&dst_batch, &unmap_batch, nr_batch);
>>>>
>>>> retry = 1;
>>>> for (pass = 0; pass < nr_pass && retry; pass++) {
>>>
>>> Found a difference between v5 and v6 here. In v5, migrate_pages_batch()
>>> checks the return value of migrate_offload_batch_copy() and acts
>>> accordingly. While in v6, it does not. Why was this changed?
>>>
>>> IIUC, FOLIO_CONTENT_COPIED is now used to indicate a copy failure. Is
>>> this necessary? Is it possible that some (but not all) folios fail to
>>> be copied?
>>
>> Based on the discussion[1], FOLIO_CONTENT_COPIED is used to indicate both
>> 1. folio is copied via offloading, and 2. MC copy failure. So the code
>> no longer needs to store the return value of migrate_offload_batch_copy()
>> and can just check dst->migrate_info & FOLIO_CONTENT_COPIED.
>>
>> For 32bit, FOLIO_CONTENT_COPIED is always 0, so all folios are always
>> copied in the serialized path. Hmm, that might be causing double copying,
>> since migrate_folios_mc_copy() copies batched folios and later
>> migrate_folios_move() copies them again since
>> dst->migrate_info & FOLIO_CONTENT_COPIED is always false. Maybe we could
>> make migrate_folios_mc_copy() return immediately in 32bit, otherwise,
>> we will need to find somewhere else to store FOLIO_CONTENT_COPIED.
>>
>> [1] https://lore.kernel.org/all/c2bfdf39-8caa-43bd-b1ad-2285b4cc767a@amd.com/
>>
>>>
> It is not cause double copying.
> migrate_should_offload() /and migrate_offload_enabled
> gate if folio can be added to unmap_batch/dst_batch.
So migrate_should_offload() is actually migrate_should_offload_batched_copy().
32BIT does not get batched copy nor offloaded copy.
>
> migrate_offload_enabled branch is enabled from the offload driver like DCBM (which
> again depends on 64_BIT) will call migrate_offload_register().
>
> I think I can add a defensive 64_BIT check in migrate_should_offload() to prevent
> any potential bugs.
I think a better name would be helpful and sufficient.
BTW, ideally, batched copy and offloaded copy are two separate features
and could be enabled separately. But considering the hassle of finding a
place to store FOLIO_CONTENT_COPIED on 32BIT, it might be fine to
have a combined batched and offloaded copy.
Best Regards,
Yan, Zi
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH RFC v6 3/5] mm/migrate: add copy offload registration infrastructure
2026-07-20 14:44 ` Zi Yan
2026-07-20 15:32 ` Garg, Shivank
@ 2026-07-21 11:32 ` Huang, Ying
1 sibling, 0 replies; 12+ messages in thread
From: Huang, Ying @ 2026-07-21 11:32 UTC (permalink / raw)
To: Zi Yan
Cc: Shivank Garg, Andrew Morton, David Hildenbrand, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price,
Alistair Popple, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko,
Karim Manaouil, Frank van der Linden, Teja Vojjala,
Pravin Tamkhane, Kinsey Ho, Wei Xu, Matthew Wilcox,
Davidlohr Bueso, Vinod Koul, Bharata B Rao, SeongJae Park,
David Rientjes, Xuezheng Chu, Yiannis Nikolakopoulos, Dave Hansen,
Johannes Weiner, John Hubbard, Peter Xu, Rik van Riel,
Shakeel Butt, Tejun Heo, Fan Ni, Jonathan Cameron,
Aneesh Kumar K.V, Nathan Lynch, Frank Li, Dan Williams, linux-mm,
linux-kernel, Mike Day
Zi Yan <ziy@nvidia.com> writes:
> On 20 Jul 2026, at 7:19, Huang, Ying wrote:
>
>> Shivank Garg <shivankg@amd.com> writes:
>>
>> [snip]
>>
>>> diff --git a/mm/migrate.c b/mm/migrate.c
>>> index 41b732e78a67..4fed3110ca0a 100644
>>> --- a/mm/migrate.c
>>> +++ b/mm/migrate.c
>>> @@ -43,6 +43,7 @@
>>> #include <linux/sched/sysctl.h>
>>> #include <linux/memory-tiers.h>
>>> #include <linux/pagewalk.h>
>>> +#include <linux/migrate_copy_offload.h>
>>>
>>> #include <asm/tlbflush.h>
>>>
>>> @@ -51,12 +52,6 @@
>>> #include "internal.h"
>>> #include "swap.h"
>>>
>>> -/* For now, never offload. Wired up in later patch. */
>>> -static bool migrate_should_offload(int reason)
>>> -{
>>> - return false;
>>> -}
>>> -
>>> static const struct movable_operations *offline_movable_ops;
>>> static const struct movable_operations *zsmalloc_movable_ops;
>>>
>>> @@ -2050,7 +2045,7 @@ static int migrate_pages_batch(struct list_head *from,
>>>
>>> /* Batch-copy eligible folios before the move phase */
>>> if (!list_empty(&unmap_batch))
>>> - migrate_folios_mc_copy(&dst_batch, &unmap_batch, nr_batch);
>>> + migrate_offload_batch_copy(&dst_batch, &unmap_batch, nr_batch);
>>>
>>> retry = 1;
>>> for (pass = 0; pass < nr_pass && retry; pass++) {
>>
>> Found a difference between v5 and v6 here. In v5, migrate_pages_batch()
>> checks the return value of migrate_offload_batch_copy() and acts
>> accordingly. While in v6, it does not. Why was this changed?
>>
>> IIUC, FOLIO_CONTENT_COPIED is now used to indicate a copy failure. Is
>> this necessary? Is it possible that some (but not all) folios fail to
>> be copied?
>
> Based on the discussion[1], FOLIO_CONTENT_COPIED is used to indicate both
> 1. folio is copied via offloading, and 2. MC copy failure. So the code
> no longer needs to store the return value of migrate_offload_batch_copy()
> and can just check dst->migrate_info & FOLIO_CONTENT_COPIED.
>
> For 32bit, FOLIO_CONTENT_COPIED is always 0, so all folios are always
> copied in the serialized path. Hmm, that might be causing double copying,
> since migrate_folios_mc_copy() copies batched folios and later
> migrate_folios_move() copies them again since
> dst->migrate_info & FOLIO_CONTENT_COPIED is always false. Maybe we could
> make migrate_folios_mc_copy() return immediately in 32bit, otherwise,
> we will need to find somewhere else to store FOLIO_CONTENT_COPIED.
>
> [1] https://lore.kernel.org/all/c2bfdf39-8caa-43bd-b1ad-2285b4cc767a@amd.com/
Thanks a lot for your information! Although personally I don't think
that it's necessary to optimize for really rare folio_mc_copy() failure
cases, I am fine with it if you think that this is the better way.
>>
>> [snip]
>>
---
Best Regards,
Huang, Ying
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH RFC v6 4/5] drivers/migrate_offload: add DMA batch copy driver (dcbm)
2026-06-30 7:28 [PATCH RFC v6 0/5] Accelerate page migration with batch copying and hardware offload Shivank Garg
` (2 preceding siblings ...)
2026-06-30 7:28 ` [PATCH RFC v6 3/5] mm/migrate: add copy offload registration infrastructure Shivank Garg
@ 2026-06-30 7:28 ` Shivank Garg
2026-06-30 7:28 ` [PATCH RFC v6 5/5] mm/migrate: adjust NR_MAX_BATCHED_MIGRATION for testing Shivank Garg
2026-07-14 11:29 ` [PATCH RFC v6 0/5] Accelerate page migration with batch copying and hardware offload Huang, Ying
5 siblings, 0 replies; 12+ messages in thread
From: Shivank Garg @ 2026-06-30 7:28 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Zi Yan, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price, Ying Huang,
Alistair Popple, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko
Cc: Karim Manaouil, Frank van der Linden, Teja Vojjala,
Pravin Tamkhane, Kinsey Ho, Wei Xu, Matthew Wilcox,
Davidlohr Bueso, Vinod Koul, Bharata B Rao, SeongJae Park,
David Rientjes, Xuezheng Chu, Yiannis Nikolakopoulos, Dave Hansen,
Johannes Weiner, John Hubbard, Peter Xu, Rik van Riel,
Shakeel Butt, Tejun Heo, Fan Ni, Jonathan Cameron,
Aneesh Kumar K.V, Nathan Lynch, Frank Li, Dan Williams, linux-mm,
linux-kernel, Shivank Garg
Add a simple DMAEngine-based migrator that plugs into the page
migration copy-offload infrastructure and batch-copies folios via
DMA memcpy channels. It is intended for testing the offload plumbing
and as a template for future migrators (SDXI, multi-threaded CPU copy,
etc.).
On success, the dst folios are marked with FOLIO_CONTENT_COPIED so
__migrate_folio skips the per-folio copy. On any DMA error,
FOLIO_CONTENT_COPIED market is not set and the migration falls back
to per-folio CPU copy.
dcbm registers with reason mask with reason mask
MIGRATE_OFFLOAD_REASONS_ALLOWED, this set can be narrowed at runtime
via the reason_mask parameter.
Runtime controls are under /sys/module/dcbm/parameters/:
offloading - enable/disable DMA offload (0/1)
nr_dma_chan - max DMA channels to use (1..N)
reason_mask - reasons to offload: names (e.g. "compaction,demotion"),
"all"/"none" or raw numerical mask.
folios_migrated - folios DMA-copied (write to reset)
folios_failures - fallback count (write to reset)
CONFIG_DCBM_DMA selects MIGRATION_COPY_OFFLOAD so enabling the
driver pulls in the infrastructure automatically.
Channel acquisition uses dma_request_chan_by_mask(DMA_MEMCPY), which
works for providers that set DMA_PRIVATE (e.g. AMD PTDMA). Generic
mem-to-mem engines that do not set DMA_PRIVATE (e.g. SDXI) should
acquire channels via dma_find_channel(DMA_MEMCPY) or the async_tx
APIs, which can be added in a follow-up.
[Correctness Note:
Karim: Descriptor can complete out of order for some dmaengines.
so descriptor chaining may yield incorrect results. For correctness
we need to add callback to every descriptor
https://lore.kernel.org/all/20260619160725.lfcxrbj5go67qy6u@wrangler
Shivank: device_prep_dma_memcpy_sg() would solve this.
]
Reviewed-by: Karim Manaouil <kmanaouil.dev@gmail.com>
Signed-off-by: Shivank Garg <shivankg@amd.com>
---
drivers/Kconfig | 2 +
drivers/Makefile | 2 +
drivers/migrate_offload/Kconfig | 9 +
drivers/migrate_offload/Makefile | 1 +
drivers/migrate_offload/dcbm/Makefile | 1 +
drivers/migrate_offload/dcbm/dcbm.c | 481 ++++++++++++++++++++++++++++++++++
6 files changed, 496 insertions(+)
diff --git a/drivers/Kconfig b/drivers/Kconfig
index f2bed2ddeb66..3e83a1475cbc 100644
--- a/drivers/Kconfig
+++ b/drivers/Kconfig
@@ -253,4 +253,6 @@ source "drivers/cdx/Kconfig"
source "drivers/resctrl/Kconfig"
+source "drivers/migrate_offload/Kconfig"
+
endmenu
diff --git a/drivers/Makefile b/drivers/Makefile
index 0841ea851847..88cb8e3e88df 100644
--- a/drivers/Makefile
+++ b/drivers/Makefile
@@ -42,6 +42,8 @@ obj-y += clk/
# really early.
obj-$(CONFIG_DMADEVICES) += dma/
+obj-$(CONFIG_MIGRATION_COPY_OFFLOAD) += migrate_offload/
+
# SOC specific infrastructure drivers.
obj-y += soc/
obj-$(CONFIG_PM_GENERIC_DOMAINS) += pmdomain/
diff --git a/drivers/migrate_offload/Kconfig b/drivers/migrate_offload/Kconfig
new file mode 100644
index 000000000000..c9f2e21a95cf
--- /dev/null
+++ b/drivers/migrate_offload/Kconfig
@@ -0,0 +1,9 @@
+config DCBM_DMA
+ tristate "DMA Core Batch Migrator"
+ depends on MIGRATION && DMA_ENGINE && 64BIT
+ select MIGRATION_COPY_OFFLOAD
+ help
+ DMA-based batch copy engine for page migration. Uses
+ DMAEngine memcpy channels to offload folio data copies
+ during migration. Primarily intended for testing the copy
+ offload infrastructure.
diff --git a/drivers/migrate_offload/Makefile b/drivers/migrate_offload/Makefile
new file mode 100644
index 000000000000..9e16018beb15
--- /dev/null
+++ b/drivers/migrate_offload/Makefile
@@ -0,0 +1 @@
+obj-$(CONFIG_DCBM_DMA) += dcbm/
diff --git a/drivers/migrate_offload/dcbm/Makefile b/drivers/migrate_offload/dcbm/Makefile
new file mode 100644
index 000000000000..56ba47cce0f1
--- /dev/null
+++ b/drivers/migrate_offload/dcbm/Makefile
@@ -0,0 +1 @@
+obj-$(CONFIG_DCBM_DMA) += dcbm.o
diff --git a/drivers/migrate_offload/dcbm/dcbm.c b/drivers/migrate_offload/dcbm/dcbm.c
new file mode 100644
index 000000000000..bacbcc689171
--- /dev/null
+++ b/drivers/migrate_offload/dcbm/dcbm.c
@@ -0,0 +1,481 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * DMA Core Batch Migrator (DCBM)
+ *
+ * Uses DMAEngine memcpy channels to offload batch folio copies during
+ * page migration. Reference driver meant for testing the offload
+ * infrastructure.
+ *
+ * Copyright (C) 2024-26 Advanced Micro Devices, Inc.
+ */
+
+#include <linux/module.h>
+#include <linux/dma-mapping.h>
+#include <linux/dmaengine.h>
+#include <linux/migrate.h>
+#include <linux/migrate_copy_offload.h>
+
+#define MAX_DMA_CHANNELS 16
+
+static atomic_long_t folios_migrated;
+static atomic_long_t folios_failures;
+
+static bool offloading_enabled;
+static unsigned int nr_dma_channels = 1;
+static DEFINE_MUTEX(dcbm_mutex);
+
+struct dma_work {
+ struct dma_chan *chan;
+ struct completion done;
+ atomic_t pending;
+ struct sg_table *src_sgt;
+ struct sg_table *dst_sgt;
+ bool mapped;
+};
+
+static void dma_completion_callback(void *data)
+{
+ struct dma_work *work = data;
+
+ if (atomic_dec_and_test(&work->pending))
+ complete(&work->done);
+}
+
+static int setup_sg_tables(struct dma_work *work, struct list_head **src_pos,
+ struct list_head **dst_pos, int nr)
+{
+ struct scatterlist *sg_src, *sg_dst;
+ struct device *dev;
+ int i, ret;
+
+ work->src_sgt = kmalloc_obj(*work->src_sgt, GFP_KERNEL);
+ if (!work->src_sgt)
+ return -ENOMEM;
+ work->dst_sgt = kmalloc_obj(*work->dst_sgt, GFP_KERNEL);
+ if (!work->dst_sgt) {
+ ret = -ENOMEM;
+ goto err_free_src;
+ }
+
+ ret = sg_alloc_table(work->src_sgt, nr, GFP_KERNEL);
+ if (ret)
+ goto err_free_dst;
+ ret = sg_alloc_table(work->dst_sgt, nr, GFP_KERNEL);
+ if (ret)
+ goto err_free_src_table;
+
+ sg_src = work->src_sgt->sgl;
+ sg_dst = work->dst_sgt->sgl;
+ for (i = 0; i < nr; i++) {
+ struct folio *src = list_entry(*src_pos, struct folio, lru);
+ struct folio *dst = list_entry(*dst_pos, struct folio, lru);
+
+ sg_set_folio(sg_src, src, folio_size(src), 0);
+ sg_set_folio(sg_dst, dst, folio_size(dst), 0);
+
+ *src_pos = (*src_pos)->next;
+ *dst_pos = (*dst_pos)->next;
+
+ if (i < nr - 1) {
+ sg_src = sg_next(sg_src);
+ sg_dst = sg_next(sg_dst);
+ }
+ }
+
+ dev = dmaengine_get_dma_device(work->chan);
+ if (!dev) {
+ ret = -ENODEV;
+ goto err_free_dst_table;
+ }
+ ret = dma_map_sgtable(dev, work->src_sgt, DMA_TO_DEVICE,
+ DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_NO_KERNEL_MAPPING);
+ if (ret)
+ goto err_free_dst_table;
+ ret = dma_map_sgtable(dev, work->dst_sgt, DMA_FROM_DEVICE,
+ DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_NO_KERNEL_MAPPING);
+ if (ret)
+ goto err_unmap_src;
+
+ /*
+ * TODO: IOMMU may merge segments unevenly on the two sides, fall back
+ * bail to CPU copy. In practice, I have not observed merging in tests.
+ * Handling unequal nents is left for follow-up.
+ */
+ if (work->src_sgt->nents != work->dst_sgt->nents) {
+ ret = -EINVAL;
+ goto err_unmap_dst;
+ }
+ work->mapped = true;
+ return 0;
+
+err_unmap_dst:
+ dma_unmap_sgtable(dev, work->dst_sgt, DMA_FROM_DEVICE,
+ DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_NO_KERNEL_MAPPING);
+err_unmap_src:
+ dma_unmap_sgtable(dev, work->src_sgt, DMA_TO_DEVICE,
+ DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_NO_KERNEL_MAPPING);
+err_free_dst_table:
+ sg_free_table(work->dst_sgt);
+err_free_src_table:
+ sg_free_table(work->src_sgt);
+err_free_dst:
+ kfree(work->dst_sgt);
+ work->dst_sgt = NULL;
+err_free_src:
+ kfree(work->src_sgt);
+ work->src_sgt = NULL;
+ return ret;
+}
+
+static void cleanup_dma_work(struct dma_work *works, int actual_channels)
+{
+ struct device *dev;
+ int i;
+
+ if (!works)
+ return;
+
+ for (i = 0; i < actual_channels; i++) {
+ if (!works[i].chan)
+ continue;
+
+ dev = dmaengine_get_dma_device(works[i].chan);
+
+ if (works[i].mapped)
+ dmaengine_terminate_sync(works[i].chan);
+
+ if (dev && works[i].mapped) {
+ if (works[i].src_sgt) {
+ dma_unmap_sgtable(dev, works[i].src_sgt,
+ DMA_TO_DEVICE,
+ DMA_ATTR_SKIP_CPU_SYNC |
+ DMA_ATTR_NO_KERNEL_MAPPING);
+ sg_free_table(works[i].src_sgt);
+ kfree(works[i].src_sgt);
+ }
+ if (works[i].dst_sgt) {
+ dma_unmap_sgtable(dev, works[i].dst_sgt,
+ DMA_FROM_DEVICE,
+ DMA_ATTR_SKIP_CPU_SYNC |
+ DMA_ATTR_NO_KERNEL_MAPPING);
+ sg_free_table(works[i].dst_sgt);
+ kfree(works[i].dst_sgt);
+ }
+ }
+ dma_release_channel(works[i].chan);
+ }
+ kfree(works);
+}
+
+static int submit_dma_transfers(struct dma_work *work)
+{
+ struct scatterlist *sg_src, *sg_dst;
+ struct dma_async_tx_descriptor *tx;
+ unsigned long flags = DMA_CTRL_ACK;
+ dma_cookie_t cookie;
+ int i;
+
+ atomic_set(&work->pending, 1);
+
+ sg_src = work->src_sgt->sgl;
+ sg_dst = work->dst_sgt->sgl;
+ for_each_sgtable_dma_sg(work->src_sgt, sg_src, i) {
+ if (i == work->src_sgt->nents - 1)
+ flags |= DMA_PREP_INTERRUPT;
+
+ tx = dmaengine_prep_dma_memcpy(work->chan,
+ sg_dma_address(sg_dst),
+ sg_dma_address(sg_src),
+ sg_dma_len(sg_src), flags);
+ if (!tx) {
+ atomic_set(&work->pending, 0);
+ return -EIO;
+ }
+
+ if (i == work->src_sgt->nents - 1) {
+ tx->callback = dma_completion_callback;
+ tx->callback_param = work;
+ }
+
+ cookie = dmaengine_submit(tx);
+ if (dma_submit_error(cookie)) {
+ atomic_set(&work->pending, 0);
+ return -EIO;
+ }
+ sg_dst = sg_next(sg_dst);
+ }
+ return 0;
+}
+
+/**
+ * folios_copy_dma - copy a batch of folios via DMA memcpy
+ * @dst_list: destination folio list
+ * @src_list: source folio list
+ * @nr_folios: number of folios in each list
+ *
+ * Return: 0 on success, negative errno on failure.
+ */
+static int folios_copy_dma(struct list_head *dst_list,
+ struct list_head *src_list, unsigned int nr_folios)
+{
+ struct folio *dst;
+ struct dma_work *works;
+ struct list_head *src_pos = src_list->next;
+ struct list_head *dst_pos = dst_list->next;
+ int i, folios_per_chan, ret;
+ dma_cap_mask_t mask;
+ int actual_channels = 0;
+ unsigned int max_channels;
+
+ max_channels = min3(READ_ONCE(nr_dma_channels), nr_folios,
+ (unsigned int)MAX_DMA_CHANNELS);
+
+ works = kcalloc(max_channels, sizeof(*works), GFP_KERNEL);
+ if (!works)
+ return -ENOMEM;
+
+ dma_cap_zero(mask);
+ dma_cap_set(DMA_MEMCPY, mask);
+
+ for (i = 0; i < max_channels; i++) {
+ works[actual_channels].chan = dma_request_chan_by_mask(&mask);
+ if (IS_ERR(works[actual_channels].chan))
+ break;
+ init_completion(&works[actual_channels].done);
+ actual_channels++;
+ }
+
+ if (actual_channels == 0) {
+ kfree(works);
+ return -ENODEV;
+ }
+
+ for (i = 0; i < actual_channels; i++) {
+ folios_per_chan = nr_folios * (i + 1) / actual_channels -
+ (nr_folios * i) / actual_channels;
+ if (folios_per_chan == 0)
+ continue;
+
+ ret = setup_sg_tables(&works[i], &src_pos, &dst_pos,
+ folios_per_chan);
+ if (ret)
+ goto err_cleanup;
+ }
+
+ for (i = 0; i < actual_channels; i++) {
+ if (!works[i].mapped)
+ continue;
+ ret = submit_dma_transfers(&works[i]);
+ if (ret)
+ goto err_cleanup;
+ }
+
+ for (i = 0; i < actual_channels; i++) {
+ if (atomic_read(&works[i].pending) > 0)
+ dma_async_issue_pending(works[i].chan);
+ }
+
+ for (i = 0; i < actual_channels; i++) {
+ if (atomic_read(&works[i].pending) == 0)
+ continue;
+ if (!wait_for_completion_timeout(&works[i].done,
+ msecs_to_jiffies(10000))) {
+ ret = -ETIMEDOUT;
+ goto err_cleanup;
+ }
+ }
+
+ /*
+ * All folios copied; mark each dst with FOLIO_CONTENT_COPIED so
+ * __migrate_folio() skips the per-folio copy in the move phase.
+ */
+ list_for_each_entry(dst, dst_list, lru)
+ dst->migrate_info |= FOLIO_CONTENT_COPIED;
+
+ cleanup_dma_work(works, actual_channels);
+
+ atomic_long_add(nr_folios, &folios_migrated);
+ return 0;
+
+err_cleanup:
+ pr_warn_ratelimited("dcbm: DMA copy failed (%d), falling back to CPU\n",
+ ret);
+ cleanup_dma_work(works, actual_channels);
+
+ atomic_long_add(nr_folios, &folios_failures);
+ return ret;
+}
+
+static const struct migrator dma_migrator = {
+ .name = "DCBM",
+ .offload_copy = folios_copy_dma,
+ .owner = THIS_MODULE,
+};
+
+static unsigned long dcbm_reason_mask = MIGRATE_OFFLOAD_REASONS_ALLOWED;
+
+/* offloading: enable/disable DMA migration offload */
+static int offloading_param_set(const char *val, const struct kernel_param *kp)
+{
+ bool enable;
+ int ret;
+
+ ret = kstrtobool(val, &enable);
+ if (ret)
+ return ret;
+
+ mutex_lock(&dcbm_mutex);
+ if (enable == offloading_enabled) {
+ mutex_unlock(&dcbm_mutex);
+ return 0;
+ }
+ if (enable) {
+ ret = migrate_offload_register(&dma_migrator,
+ READ_ONCE(dcbm_reason_mask));
+ if (ret) {
+ mutex_unlock(&dcbm_mutex);
+ return ret;
+ }
+ WRITE_ONCE(offloading_enabled, true);
+ } else {
+ migrate_offload_unregister(&dma_migrator);
+ WRITE_ONCE(offloading_enabled, false);
+ }
+ mutex_unlock(&dcbm_mutex);
+ return 0;
+}
+
+static int offloading_param_get(char *buffer, const struct kernel_param *kp)
+{
+ return sysfs_emit(buffer, "%d\n", READ_ONCE(offloading_enabled));
+}
+
+static const struct kernel_param_ops offloading_param_ops = {
+ .set = offloading_param_set,
+ .get = offloading_param_get,
+};
+module_param_cb(offloading, &offloading_param_ops, NULL, 0644);
+MODULE_PARM_DESC(offloading, "Enable DMA migration offload (0/1)");
+
+/* nr_dma_chan: max DMA channels to use per batch */
+static int nr_dma_chan_param_set(const char *val, const struct kernel_param *kp)
+{
+ unsigned int new_val;
+ int ret;
+
+ ret = kstrtouint(val, 0, &new_val);
+ if (ret)
+ return ret;
+ if (new_val < 1 || new_val > MAX_DMA_CHANNELS)
+ return -EINVAL;
+
+ mutex_lock(&dcbm_mutex);
+ WRITE_ONCE(nr_dma_channels, new_val);
+ mutex_unlock(&dcbm_mutex);
+ return 0;
+}
+
+static int nr_dma_chan_param_get(char *buffer, const struct kernel_param *kp)
+{
+ return sysfs_emit(buffer, "%u\n", READ_ONCE(nr_dma_channels));
+}
+
+static const struct kernel_param_ops nr_dma_chan_param_ops = {
+ .set = nr_dma_chan_param_set,
+ .get = nr_dma_chan_param_get,
+};
+module_param_cb(nr_dma_chan, &nr_dma_chan_param_ops, NULL, 0644);
+MODULE_PARM_DESC(nr_dma_chan, "Max DMA channels to use (1..16)");
+
+/* reason_mask: set of MR_* reasons this migrator handles */
+static int reason_mask_param_set(const char *val, const struct kernel_param *kp)
+{
+ unsigned long mask;
+ int ret;
+
+ ret = migrate_offload_reason_mask_parse(val, &mask);
+ if (ret)
+ return ret;
+
+ mutex_lock(&dcbm_mutex);
+ WRITE_ONCE(dcbm_reason_mask, mask);
+ if (offloading_enabled)
+ migrate_offload_set_reason_mask(&dma_migrator, mask);
+ mutex_unlock(&dcbm_mutex);
+ return 0;
+}
+
+static int reason_mask_param_get(char *buffer, const struct kernel_param *kp)
+{
+ return migrate_offload_reason_mask_format(buffer, READ_ONCE(dcbm_reason_mask));
+}
+
+static const struct kernel_param_ops reason_mask_param_ops = {
+ .set = reason_mask_param_set,
+ .get = reason_mask_param_get,
+};
+module_param_cb(reason_mask, &reason_mask_param_ops, NULL, 0644);
+MODULE_PARM_DESC(reason_mask,
+ "Reasons to offload: comma-separated names (e.g. compaction,demotion), 'all', 'none', or a raw hex mask");
+
+/* folios_migrated / folios_failures: counters; any write resets to 0 */
+static int folios_migrated_param_set(const char *val, const struct kernel_param *kp)
+{
+ atomic_long_set(&folios_migrated, 0);
+ return 0;
+}
+
+static int folios_migrated_param_get(char *buffer, const struct kernel_param *kp)
+{
+ return sysfs_emit(buffer, "%ld\n", atomic_long_read(&folios_migrated));
+}
+
+static const struct kernel_param_ops folios_migrated_param_ops = {
+ .set = folios_migrated_param_set,
+ .get = folios_migrated_param_get,
+};
+module_param_cb(folios_migrated, &folios_migrated_param_ops, NULL, 0644);
+MODULE_PARM_DESC(folios_migrated, "Folios DMA-copied (write to reset)");
+
+static int folios_failures_param_set(const char *val, const struct kernel_param *kp)
+{
+ atomic_long_set(&folios_failures, 0);
+ return 0;
+}
+
+static int folios_failures_param_get(char *buffer, const struct kernel_param *kp)
+{
+ return sysfs_emit(buffer, "%ld\n", atomic_long_read(&folios_failures));
+}
+
+static const struct kernel_param_ops folios_failures_param_ops = {
+ .set = folios_failures_param_set,
+ .get = folios_failures_param_get,
+};
+module_param_cb(folios_failures, &folios_failures_param_ops, NULL, 0644);
+MODULE_PARM_DESC(folios_failures, "DMA-copy failure count (write to reset)");
+
+static int __init dcbm_init(void)
+{
+ pr_info("dcbm: DMA Core Batch Migrator initialized\n");
+ return 0;
+}
+
+static void __exit dcbm_exit(void)
+{
+ mutex_lock(&dcbm_mutex);
+ if (offloading_enabled) {
+ migrate_offload_unregister(&dma_migrator);
+ offloading_enabled = false;
+ }
+ mutex_unlock(&dcbm_mutex);
+
+ pr_info("dcbm: DMA Core Batch Migrator unloaded\n");
+}
+
+module_init(dcbm_init);
+module_exit(dcbm_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Shivank Garg");
+MODULE_DESCRIPTION("DMA Core Batch Migrator");
--
2.43.0
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH RFC v6 5/5] mm/migrate: adjust NR_MAX_BATCHED_MIGRATION for testing
2026-06-30 7:28 [PATCH RFC v6 0/5] Accelerate page migration with batch copying and hardware offload Shivank Garg
` (3 preceding siblings ...)
2026-06-30 7:28 ` [PATCH RFC v6 4/5] drivers/migrate_offload: add DMA batch copy driver (dcbm) Shivank Garg
@ 2026-06-30 7:28 ` Shivank Garg
2026-07-14 11:29 ` [PATCH RFC v6 0/5] Accelerate page migration with batch copying and hardware offload Huang, Ying
5 siblings, 0 replies; 12+ messages in thread
From: Shivank Garg @ 2026-06-30 7:28 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Zi Yan, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price, Ying Huang,
Alistair Popple, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko
Cc: Karim Manaouil, Frank van der Linden, Teja Vojjala,
Pravin Tamkhane, Kinsey Ho, Wei Xu, Matthew Wilcox,
Davidlohr Bueso, Vinod Koul, Bharata B Rao, SeongJae Park,
David Rientjes, Xuezheng Chu, Yiannis Nikolakopoulos, Dave Hansen,
Johannes Weiner, John Hubbard, Peter Xu, Rik van Riel,
Shakeel Butt, Tejun Heo, Fan Ni, Jonathan Cameron,
Aneesh Kumar K.V, Nathan Lynch, Frank Li, Dan Williams, linux-mm,
linux-kernel, Shivank Garg
From: Zi Yan <ziy@nvidia.com>
Change NR_MAX_BATCHED_MIGRATION to HPAGE_PUD_NR to allow batching THP
copies.
These are for testing purpose only.
Signed-off-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Shivank Garg <shivankg@amd.com>
---
mm/migrate.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/mm/migrate.c b/mm/migrate.c
index 4fed3110ca0a..4be1f3dc1603 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -1592,7 +1592,7 @@ static inline int try_split_folio(struct folio *folio, struct list_head *split_f
}
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
-#define NR_MAX_BATCHED_MIGRATION HPAGE_PMD_NR
+#define NR_MAX_BATCHED_MIGRATION HPAGE_PUD_NR
#else
#define NR_MAX_BATCHED_MIGRATION 512
#endif
--
2.43.0
^ permalink raw reply related [flat|nested] 12+ messages in thread* Re: [PATCH RFC v6 0/5] Accelerate page migration with batch copying and hardware offload
2026-06-30 7:28 [PATCH RFC v6 0/5] Accelerate page migration with batch copying and hardware offload Shivank Garg
` (4 preceding siblings ...)
2026-06-30 7:28 ` [PATCH RFC v6 5/5] mm/migrate: adjust NR_MAX_BATCHED_MIGRATION for testing Shivank Garg
@ 2026-07-14 11:29 ` Huang, Ying
5 siblings, 0 replies; 12+ messages in thread
From: Huang, Ying @ 2026-07-14 11:29 UTC (permalink / raw)
To: Shivank Garg
Cc: Andrew Morton, David Hildenbrand, Zi Yan, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price,
Alistair Popple, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko,
Karim Manaouil, Frank van der Linden, Teja Vojjala,
Pravin Tamkhane, Kinsey Ho, Wei Xu, Matthew Wilcox,
Davidlohr Bueso, Vinod Koul, Bharata B Rao, SeongJae Park,
David Rientjes, Xuezheng Chu, Yiannis Nikolakopoulos, Dave Hansen,
Johannes Weiner, John Hubbard, Peter Xu, Rik van Riel,
Shakeel Butt, Tejun Heo, Fan Ni, Jonathan Cameron,
Aneesh Kumar K.V, Nathan Lynch, Frank Li, Dan Williams, linux-mm,
linux-kernel, Mike Day
Hi, Garg,
Thanks for updated patch.
Shivank Garg <shivankg@amd.com> writes:
[snip]
>
> PERFORMANCE RESULTS:
> --------------------
>
> AMD EPYC 7713 (Zen 3), 2 sockets, 32 cores, SMT on,
> 1 NUMA node per socket, 256 GB/node, v7.2-rc1, DVFS=Performance, PTDMA
> (16 DMA channels).
>
> Benchmark: move_pages() syscall to move pages between two NUMA nodes.
>
> 1). Moving different sized folios such that total transfer size is constant
> (1GB), with different number of DMA channels. Throughput in GB/s.
>
> a. Baseline (vanilla kernel, single-threaded, serial folio_copy):
> ================================================================================
> 4K | 16K | 64K | 256K | 1M | 2M |
> ================================================================================
> 3.28±0.14 | 4.98±0.18 | 6.19±0.08 | 6.77±0.08 | 7.02±0.11 | 10.80±0.13 |
>
> b. DMA offload (Patched Kernel, dcbm driver, N DMA channels):
> ============================================================================================
> N channel| 4K | 16K | 64K | 256K | 1M | 2M |
> ============================================================================================
> 1 | 2.38±0.17 | 2.77±0.03 | 3.21±0.03 | 5.00±0.02 | 5.09±0.64 | 12.62±0.07 |
> 2 | 2.87±0.11 | 4.06±0.05 | 5.09±0.04 | 6.97±0.08 | 8.43±0.06 | 14.32±0.10 |
> 4 | 3.32±0.07 | 5.30±0.06 | 7.21±0.09 | 9.69±0.15 | 11.36±0.13 | 26.98±0.19 |
> 8 | 3.68±0.09 | 6.28±0.10 | 9.16±0.13 | 12.05±0.16 | 15.33±2.80 | 46.06±0.55 |
> 12 | 3.83±0.05 | 6.65±0.17 | 10.00±0.16 | 12.98±0.18 | 15.87±0.19 | 61.31±1.28 |
> 16 | 3.94±0.09 | 6.78±0.10 | 10.48±0.13 | 13.48±0.20 | 16.90±0.24 | 65.06±2.46 |
>
> 2). First-folio latency: custom tracepoints (in migrate_pages_batch enter/exit,
> migrate_folio_done) measure latency per migrate_pages_batch() call.
>
> Throughput (GB/s) and first-folio latency (us), median of 10 runs.
>
> a. Vanilla Kernel:
>
> NR_MAX_BATCHED_MIGRATION upstream default value is 512.
> --- Order 0 (4K folios) --- --- Order 9 (2M folios) ---
> n vanilla/cpu n vanilla/cpu
> (folios) GB/s | first(us) (folios) GB/s | first(us)
> -------------------------- --------------------------
> 1 0.03 | 24 1 6.86 | 204
> 4 0.13 | 30 4 8.68 | 191
> 8 0.27 | 27 8 7.92 | 207
> 16 0.43 | 34 16 6.77 | 234
> 64 1.12 | 51 64 10.44 | 179
> 256 1.67 | 166 256 10.43 | 181
> 512 1.98 | 255 512 10.55 | 179
> 2048 2.38 | 233
> 4096 2.42 | 168
> 16384 2.72 | 167
> 65536 3.00 | 156
> 262144 3.10 | 151
>
> b. Patched kernel:
> N = NR_MAX_BATCHED_MIGRATION (in pages), Total migrated data fixed at
> 1 GB. Change N with knob (just for testing) to measure impact of
> different max batched size.
>
> --- ORDER 0 (4K folios) ---
>
> N offload/dma1 offload/dma4 offload/dma16
> GB/s | first(us) GB/s | first(us) GB/s | first(us)
> ------------------------------------------------------------------------
> 512 2.21 | 628 3.29 | 275 3.25 | 245
> 1024 2.06 | 1271 3.21 | 601 3.36 | 518
> 2048 2.02 | 2646 3.00 | 1388 3.20 | 1110
> 4096 2.08 | 4832 3.17 | 2514 3.41 | 2175
> 8192 2.16 | 9253 3.14 | 4839 3.62 | 3592
> 16384 2.24 | 17543 3.23 | 9680 3.58 | 7144
> 32768 2.22 | 36408 3.26 | 19301 3.67 | 14524
> 65536 2.12 | 82572 3.24 | 38091 3.62 | 29835
> 131072 2.08 | 153669 3.17 | 79744 3.48 | 62157
> 262144 2.05 | 332297 2.97 | 175315 3.33 | 134774
>
> --- ORDER 9 (2M folios) ---
>
> N offload/dma1 offload/dma4 offload/dma16
> GB/s | first(us) GB/s | first(us) GB/s | first(us)
> ------------------------------------------------------------------------
> 512 11.74 | 160 11.71 | 160 11.75 | 159
> 1024 12.18 | 310 13.82 | 274 13.76 | 275
> 2048 12.39 | 612 25.55 | 290 25.69 | 289
> 4096 12.54 | 1211 26.25 | 564 42.36 | 334
> 8192 12.54 | 2421 26.82 | 1111 51.85 | 485
> 16384 12.61 | 4824 26.91 | 2209 54.26 | 925
> 32768 12.62 | 9652 27.04 | 4404 54.72 | 1942
> 65536 12.64 | 19287 26.95 | 8835 57.30 | 3535
> 131072 12.64 | 38824 26.95 | 17900 58.58 | 7747
> 262144 12.66 | 77610 26.95 | 35743 66.31 | 13801
>
> OPEN QUESTION:
> --------------
>
> The best batch size depends on the hardware, and bigger isn't always better.
> NR_MAX_BATCHED_MIGRATION decides how many pages we move at once.
> Higher batch size can help amortize the setup cost of migrator but
> increases the first-folio latency (the folio is inaccessible for this
> window).
Yes. This is an important parameter for the batched migration. We
really need more information from people who have workload information
to make a decision. Can you try to reach them?
Also, can we find a sweet point where we can archive higher throughput
without increasing latency too much (e.g., < 1ms)?
> Goals could be workload dependent, e.g. higher throughput versus
> same throughput under a bounded latency.
>
> Should this be tunable to accommodate different hardware and goals?
[snip]
---
Best Regards,
Huang, Ying
^ permalink raw reply [flat|nested] 12+ messages in thread