[PATCH RFC 00/11] mm/migrate: separate migration paths and carry migration policy

[PATCH RFC 00/11] mm/migrate: separate migration paths and carry migration policy

[Shivank Garg]

migrate_pages() handles hugetlb folios, movable_ops pages and LRU folios,
although their locking, mapping and retry requirements differ. It already
dispatches hugetlb folios to a dedicated engine but movable_ops 
pages still pass through the regular folio unmap and move machinery.

Migration policy has a separate interface problem. migrate_mode describes
blocking discipline, but it is passed independently from migrate_reason
through the core, and only migrate_mode reaches
address_space_operations->migrate_folio(). Adding another policy dimension
therefore requires more positional arguments, an overloaded mode or side
state. [3] 

This series separates the migration paths (classes) for movable_ops pages
and LRU folios, moves LRU policy into the LRU engine, and carries mode and
reason through struct migrate_control.

Thanks to Zi Yan and David Hildenbrand for the discussions and suggestions
that helped shape this series. I also used AI tools (Opus 5, GPT-5.6 Sol)
to refine my ideas, identify the edge-cases/bugs.

Old flow
========

migrate_pages(mode, reason)
  |
  +-- migrate_hugetlbs()
  |     `-- scan the mixed source list (from) on each retry
  |
  `-- split the remaining list into batches
        |
        +-- MIGRATE_ASYNC
        |     `-- migrate_pages_batch()
        |           inline unmap/retry -> flush -> move/retry
        |
        `-- MIGRATE_SYNC[_LIGHT]
              `-- migrate_pages_sync()
                   +-- try async batch pre-pass
                   `-- retry failures one at a time through
                       migrate_pages_batch()

movable_ops pages are identified by branches inside the batch unmap and
move paths.

New flow
========

migrate_pages(ctl)
  |
  +-- migrate_hugetlbs()
  |     `-- collect and retry hugetlb folios
  |
  +-- migrate_movable_ops_pages()
  |     `-- allocate, lock, invoke the movable_ops callback and retry
  |
  `-- migrate_lru_folios()
        `-- form bounded LRU batches
              `-- __migrate_lru_folios()
                    +-- async: migrate_folios_batch()
                    `-- sync:
                        +-- async batch pre-pass
                        `-- retry failures one at a time

migrate_folios_batch()
  |
  +-- migrate_folios_unmap()
  +-- try_to_unmap_flush()
  +-- migrate_folios_move()
  `-- migrate_folios_undo()

migrate_pages() now only dispatches classes. Each entry owns its lists and
retries; the LRU entry also owns batching and synchronous fallback.

Motivation
==========

This refactor makes the migration control flow easier to follow and gives
new optimizations clear boundaries and insertion points.

Separating movable_ops also prepares for memdescs, where these pages are
expected to lose their folio representation and folio->lru handoff. The
current list interface remains, but its folio dependencies are now isolated.

struct migrate_control is caller-owned, stack allocated and passed as const.
It carries mode and reason through migrate_pages() and ->migrate_folio(),
and provide place for future extension to policy. This is one-time pain of
updating callers, but doing it once avoids repeating that churn whenever
a new use case or optimization need additional policy.

Follow-on 
=========

This series establishes the class, phase and policy boundaries without
adding copy implementation. Possible follow-on use looks like:

Yiannis and Alirad's RFC adds an asynchronous non-temporal mode and updates
existing asynchronous-mode checks. [1] A separate policy field would keep
cache behavior independent of blocking discipline.

  /* migrate_pages() caller: demote_folio_list() */
  ctl.reason = MR_DEMOTION;   /* migration reason - already exists */
  ctl.mode = MIGRATE_ASYNC;   /* blocking discipline - already exists */
  ctl.cache_hint = MIGRATE_COPY_NT; /* caching intent- new policy */

  __migrate_folio(..., ctl);       /* NT/offload based on caller's hint*/
     `-> folio_mc_copy() or folio_mc_copy_nt() or ...

My batch-copy/offload series [2] can extend migrate_control to take caller's
preference of copy engines (like DMA offload, multi-threaded copy, etc.) or
batch size. Some of this may remain wishful thinking but that is what the
RFC is for :)

Behavior Changes
================

No functional change is intended for the LRU and hugetlb migration paths.

The movable_ops pass has three intentional differences:

  - movable_ops pages are attempted before LRU folios;
  - movable_ops pages no longer use the LRU asynchronous pre-pass.
    synchronous callers use their requested mode from the first attempt.
  - movable_ops callback returning -EAGAIN releases the destination, so
    retry allocates a new one.

Unmigrated pages are still returned through the original source list.

[1] https://lore.kernel.org/r/20260730-rfc-nt-demote-v2-0-452dbe3b5073@zptcorp.com
[2] https://lore.kernel.org/r/20260630-shivank-batch-migrate-offload-v6-0-da95d7e8b8a2@amd.com 
[3] https://lore.kernel.org/r/cae6ab98-3441-38bd-1e07-4586a85cdc74@google.com

---
Shivank Garg (11):
      mm/migrate: extract folio unmap phase
      mm/migrate: handle retries in migrate_folios_move()
      mm/migrate: factor out folio splitting on allocation failure
      mm/migrate: use a dedicated list for hugetlb folios
      mm/migrate: add a dedicated movable_ops migration pass
      mm/migrate: rename migrate_pages_batch() to migrate_folios_batch()
      mm/migrate: add migrate_lru_folios() entry point
      mm/migrate: move LRU batching into migrate_lru_folios()
      mm/migrate: thread migration policy through a control struct
      mm/migrate: pass migrate_control to migrate_pages()
      mm/migrate: pass migrate_control to migrate_folio()

 Documentation/filesystems/locking.rst |   2 +-
 Documentation/filesystems/vfs.rst     |   7 +-
 fs/aio.c                              |   2 +-
 fs/btrfs/disk-io.c                    |   5 +-
 fs/btrfs/inode.c                      |   6 +-
 fs/hugetlbfs/inode.c                  |   4 +-
 fs/jfs/jfs_metapage.c                 |  14 +-
 fs/nfs/internal.h                     |   2 +-
 fs/nfs/write.c                        |   8 +-
 include/linux/buffer_head.h           |   6 +-
 include/linux/fs.h                    |   6 +-
 include/linux/migrate.h               |   7 +-
 include/linux/migrate_mode.h          |  14 +
 include/linux/pagemap.h               |   2 +-
 mm/compaction.c                       |   8 +-
 mm/damon/ops-common.c                 |   7 +-
 mm/gup.c                              |   7 +-
 mm/memory-failure.c                   |   6 +-
 mm/memory_hotplug.c                   |   6 +-
 mm/mempolicy.c                        |  13 +-
 mm/migrate.c                          | 717 ++++++++++++++++++++++------------
 mm/page_alloc.c                       |   6 +-
 mm/secretmem.c                        |   3 +-
 mm/vmscan.c                           |   7 +-
 virt/kvm/guest_memfd.c                |   2 +-
 25 files changed, 563 insertions(+), 304 deletions(-)
---
base-commit: cee9395acd8043be0644b25c34bfa86623f2b935
change-id: 20260824-migrate-refactor-shivank-4ee3949fcdcb

Best regards,
-- 
Shivank Garg <shivankg@amd.com>

[PATCH RFC 01/11] mm/migrate: extract folio unmap phase

[Shivank Garg]

migrate_pages_batch() was hard to follow with the unmap retry handling
inline alongside TLB flush, move, and cleanup. Extract the unmap phase
into migrate_folios_unmap(), which fills @unmap_folios and @dst_folios
for the steps that follow. This makes the unmap phase boundaries clear
and the code more readable.

No functional change intended.

Suggested-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 mm/migrate.c | 89 ++++++++++++++++++++++++++++++++++++++++--------------------
 1 file changed, 59 insertions(+), 30 deletions(-)

diff --git a/mm/migrate.c b/mm/migrate.c
index 15b45832bcfa..14a0f84009c6 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -1800,36 +1800,30 @@ static void migrate_folios_undo(struct list_head *src_folios,
 }
 
 /*
- * migrate_pages_batch() first unmaps folios in the from list as many as
- * possible, then move the unmapped folios.
+ * Unmap the folios in @from, queuing successfully unmapped sources on
+ * @unmap_folios and their destination on @dst_folios.  Increment @nr_failed for
+ * each failure.
  *
- * We only batch migration if mode == MIGRATE_ASYNC to avoid to wait a
- * lock or bit when we have locked more than one folio.  Which may cause
- * deadlock (e.g., for loop device).  So, if mode != MIGRATE_ASYNC, the
- * length of the from list must be <= 1.
+ * Return: -ENOMEM on destination allocation failure stop the unmap phase,
+ * otherwise 0.  Folios already unmapped remain queued for the move phase.
  */
-static int migrate_pages_batch(struct list_head *from,
+static int migrate_folios_unmap(struct list_head *from,
 		new_folio_t get_new_folio, free_folio_t put_new_folio,
 		unsigned long private, enum migrate_mode mode, enum migrate_reason reason,
 		struct list_head *ret_folios, struct list_head *split_folios,
-		struct migrate_pages_stats *stats, int nr_pass)
+		struct list_head *unmap_folios, struct list_head *dst_folios,
+		struct migrate_pages_stats *stats, int nr_pass, int *nr_failed)
 {
 	int retry = 1;
 	int thp_retry = 1;
-	int nr_failed = 0;
 	int nr_retry_pages = 0;
 	int pass = 0;
 	bool is_thp = false;
 	bool is_large = false;
 	struct folio *folio, *folio2, *dst = NULL;
-	int rc, rc_saved = 0, nr_pages;
-	LIST_HEAD(unmap_folios);
-	LIST_HEAD(dst_folios);
+	int rc, nr_pages;
 	bool nosplit = (reason == MR_NUMA_MISPLACED);
 
-	VM_WARN_ON_ONCE(mode != MIGRATE_ASYNC &&
-			!list_empty(from) && !list_is_singular(from));
-
 	for (pass = 0; pass < nr_pass && retry; pass++) {
 		retry = 0;
 		thp_retry = 0;
@@ -1869,7 +1863,7 @@ static int migrate_pages_batch(struct list_head *from,
 			   !list_empty(&folio->_deferred_list) &&
 			   folio_test_partially_mapped(folio)) {
 				if (!try_split_folio(folio, split_folios, mode)) {
-					nr_failed++;
+					*nr_failed += 1;
 					stats->nr_thp_failed += is_thp;
 					stats->nr_thp_split += is_thp;
 					stats->nr_split++;
@@ -1888,7 +1882,7 @@ static int migrate_pages_batch(struct list_head *from,
 			 * list is processed.
 			 */
 			if (!thp_migration_supported() && is_thp) {
-				nr_failed++;
+				*nr_failed += 1;
 				stats->nr_thp_failed++;
 				if (!try_split_folio(folio, split_folios, mode)) {
 					stats->nr_thp_split++;
@@ -1925,13 +1919,13 @@ static int migrate_pages_batch(struct list_head *from,
 			 *	-ENOMEM: stay on the from list
 			 *	Other errno: put on ret_folios list
 			 */
-			switch(rc) {
+			switch (rc) {
 			case -ENOMEM:
 				/*
 				 * When memory is low, don't bother to try to migrate
 				 * other folios, move unmapped folios, then exit.
 				 */
-				nr_failed++;
+				*nr_failed += 1;
 				stats->nr_thp_failed += is_thp;
 				/* Large folio NUMA faulting doesn't split to retry. */
 				if (is_large && !nosplit) {
@@ -1951,7 +1945,7 @@ static int migrate_pages_batch(struct list_head *from,
 						thp_retry += is_thp;
 						nr_retry_pages += nr_pages;
 						/* Undo duplicated failure counting. */
-						nr_failed--;
+						*nr_failed -= 1;
 						stats->nr_thp_failed -= is_thp;
 						break;
 					}
@@ -1960,19 +1954,15 @@ static int migrate_pages_batch(struct list_head *from,
 				stats->nr_failed_pages += nr_pages + nr_retry_pages;
 				/* nr_failed isn't updated for not used */
 				stats->nr_thp_failed += thp_retry;
-				rc_saved = rc;
-				if (list_empty(&unmap_folios))
-					goto out;
-				else
-					goto move;
+				return -ENOMEM;
 			case -EAGAIN:
 				retry++;
 				thp_retry += is_thp;
 				nr_retry_pages += nr_pages;
 				break;
 			case 0:
-				list_move_tail(&folio->lru, &unmap_folios);
-				list_add_tail(&dst->lru, &dst_folios);
+				list_move_tail(&folio->lru, unmap_folios);
+				list_add_tail(&dst->lru, dst_folios);
 				break;
 			default:
 				/*
@@ -1981,17 +1971,56 @@ static int migrate_pages_batch(struct list_head *from,
 				 * removed from migration folio list and not
 				 * retried in the next outer loop.
 				 */
-				nr_failed++;
+				*nr_failed += 1;
 				stats->nr_thp_failed += is_thp;
 				stats->nr_failed_pages += nr_pages;
 				break;
 			}
 		}
 	}
-	nr_failed += retry;
+	*nr_failed += retry;
 	stats->nr_thp_failed += thp_retry;
 	stats->nr_failed_pages += nr_retry_pages;
-move:
+
+	return 0;
+}
+
+/*
+ * migrate_pages_batch() first unmaps as many folios in the source list as
+ * possible, flushes the TLBs, then moves the unmapped folios.
+ *
+ * Only MIGRATE_ASYNC may batch multiple folios.  Waiting for a lock or bit
+ * while multiple folios are locked may deadlock (e.g. with a loop device).
+ * Therefore, if mode != MIGRATE_ASYNC, the source list must contain at most
+ * one folio.
+ */
+static int migrate_pages_batch(struct list_head *from,
+		new_folio_t get_new_folio, free_folio_t put_new_folio,
+		unsigned long private, enum migrate_mode mode, enum migrate_reason reason,
+		struct list_head *ret_folios, struct list_head *split_folios,
+		struct migrate_pages_stats *stats, int nr_pass)
+{
+	int retry = 1;
+	int thp_retry = 1;
+	int nr_failed = 0;
+	int nr_retry_pages = 0;
+	int pass = 0;
+	int rc, rc_saved;
+	LIST_HEAD(unmap_folios);
+	LIST_HEAD(dst_folios);
+
+	VM_WARN_ON_ONCE(mode != MIGRATE_ASYNC &&
+			!list_empty(from) && !list_is_singular(from));
+
+	rc_saved = migrate_folios_unmap(from, get_new_folio, put_new_folio,
+			private, mode, reason, ret_folios, split_folios,
+			&unmap_folios, &dst_folios, stats, nr_pass,
+			&nr_failed);
+	if (rc_saved && list_empty(&unmap_folios)) {
+		rc = rc_saved;
+		goto out;
+	}
+
 	/* Flush TLBs for all unmapped folios */
 	try_to_unmap_flush();
 

-- 
2.43.0

[PATCH RFC 02/11] mm/migrate: handle retries in migrate_folios_move()

[Shivank Garg]

After extracting the unmap phase, migrate_pages_batch() still drives the
retry loop for the move phase and passes its retry bookkeeping to
migrate_folios_move(). This splits the move phase across the two
functions and requires several output parameters.

Move the retry loop and its bookkeeping into migrate_folios_move().
The function now takes the number of passes and reports failures through
@nr_failed, matching migrate_folios_unmap().

No functional change is intended.

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 mm/migrate.c | 104 ++++++++++++++++++++++++++++-------------------------------
 1 file changed, 49 insertions(+), 55 deletions(-)

diff --git a/mm/migrate.c b/mm/migrate.c
index 14a0f84009c6..89207bf515b9 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -1729,51 +1729,61 @@ static void migrate_folios_move(struct list_head *src_folios,
 		free_folio_t put_new_folio, unsigned long private,
 		enum migrate_mode mode, enum migrate_reason reason,
 		struct list_head *ret_folios,
-		struct migrate_pages_stats *stats,
-		int *retry, int *thp_retry, int *nr_failed,
-		int *nr_retry_pages)
+		struct migrate_pages_stats *stats, int nr_pass,
+		int *nr_failed)
 {
 	struct folio *folio, *folio2, *dst, *dst2;
+	int retry = 1;
+	int thp_retry = 0;
+	int nr_retry_pages = 0;
 	bool is_thp;
 	int nr_pages;
-	int rc;
+	int pass, rc;
 
-	dst = list_first_entry(dst_folios, struct folio, lru);
-	dst2 = list_next_entry(dst, lru);
-	list_for_each_entry_safe(folio, folio2, src_folios, lru) {
-		is_thp = folio_test_large(folio) && folio_test_pmd_mappable(folio);
-		nr_pages = folio_nr_pages(folio);
+	for (pass = 0; pass < nr_pass && retry; pass++) {
+		retry = 0;
+		thp_retry = 0;
+		nr_retry_pages = 0;
 
-		cond_resched();
+		dst = list_first_entry(dst_folios, struct folio, lru);
+		dst2 = list_next_entry(dst, lru);
+		list_for_each_entry_safe(folio, folio2, src_folios, lru) {
+			is_thp = folio_test_large(folio) && folio_test_pmd_mappable(folio);
+			nr_pages = folio_nr_pages(folio);
 
-		rc = migrate_folio_move(put_new_folio, private,
-				folio, dst, mode,
-				reason, ret_folios);
-		/*
-		 * The rules are:
-		 *	0: folio will be freed
-		 *	-EAGAIN: stay on the src_folios list
-		 *	Other errno: put on ret_folios list
-		 */
-		switch (rc) {
-		case -EAGAIN:
-			*retry += 1;
-			*thp_retry += is_thp;
-			*nr_retry_pages += nr_pages;
-			break;
-		case 0:
-			stats->nr_succeeded += nr_pages;
-			stats->nr_thp_succeeded += is_thp;
-			break;
-		default:
-			*nr_failed += 1;
-			stats->nr_thp_failed += is_thp;
-			stats->nr_failed_pages += nr_pages;
-			break;
+			cond_resched();
+
+			rc = migrate_folio_move(put_new_folio, private,
+					folio, dst, mode, reason, ret_folios);
+			/*
+			 * The rules are:
+			 *	0: folio will be freed
+			 *	-EAGAIN: stay on the src_folios list
+			 *	Other errno: put on ret_folios list
+			 */
+			switch (rc) {
+			case -EAGAIN:
+				retry++;
+				thp_retry += is_thp;
+				nr_retry_pages += nr_pages;
+				break;
+			case 0:
+				stats->nr_succeeded += nr_pages;
+				stats->nr_thp_succeeded += is_thp;
+				break;
+			default:
+				*nr_failed += 1;
+				stats->nr_thp_failed += is_thp;
+				stats->nr_failed_pages += nr_pages;
+				break;
+			}
+			dst = dst2;
+			dst2 = list_next_entry(dst, lru);
 		}
-		dst = dst2;
-		dst2 = list_next_entry(dst, lru);
 	}
+	*nr_failed += retry;
+	stats->nr_thp_failed += thp_retry;
+	stats->nr_failed_pages += nr_retry_pages;
 }
 
 static void migrate_folios_undo(struct list_head *src_folios,
@@ -2000,11 +2010,7 @@ static int migrate_pages_batch(struct list_head *from,
 		struct list_head *ret_folios, struct list_head *split_folios,
 		struct migrate_pages_stats *stats, int nr_pass)
 {
-	int retry = 1;
-	int thp_retry = 1;
 	int nr_failed = 0;
-	int nr_retry_pages = 0;
-	int pass = 0;
 	int rc, rc_saved;
 	LIST_HEAD(unmap_folios);
 	LIST_HEAD(dst_folios);
@@ -2024,21 +2030,9 @@ static int migrate_pages_batch(struct list_head *from,
 	/* Flush TLBs for all unmapped folios */
 	try_to_unmap_flush();
 
-	retry = 1;
-	for (pass = 0; pass < nr_pass && retry; pass++) {
-		retry = 0;
-		thp_retry = 0;
-		nr_retry_pages = 0;
-
-		/* Move the unmapped folios */
-		migrate_folios_move(&unmap_folios, &dst_folios,
-				put_new_folio, private, mode, reason,
-				ret_folios, stats, &retry, &thp_retry,
-				&nr_failed, &nr_retry_pages);
-	}
-	nr_failed += retry;
-	stats->nr_thp_failed += thp_retry;
-	stats->nr_failed_pages += nr_retry_pages;
+	/* Move the unmapped folios */
+	migrate_folios_move(&unmap_folios, &dst_folios, put_new_folio, private,
+			mode, reason, ret_folios, stats, nr_pass, &nr_failed);
 
 	rc = rc_saved ? : nr_failed;
 out:

-- 
2.43.0

[PATCH RFC 03/11] mm/migrate: factor out folio splitting on allocation failure

[Shivank Garg]

If destination folio allocation fails, migrate_folios_unmap() may split
a large folio and retry migration with smaller folios. Factor this logic
out into migrate_folio_split_on_alloc_fail() to simplify the allocation
failure path.

No functional change intended.

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 mm/migrate.c | 69 +++++++++++++++++++++++++++++++++++++-----------------------
 1 file changed, 43 insertions(+), 26 deletions(-)

diff --git a/mm/migrate.c b/mm/migrate.c
index 89207bf515b9..510dfffa7235 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -1809,6 +1809,34 @@ static void migrate_folios_undo(struct list_head *src_folios,
 	}
 }
 
+static int migrate_folio_split_on_alloc_fail(struct folio *folio,
+		struct list_head *split_folios, enum migrate_mode mode,
+		enum migrate_reason reason, struct migrate_pages_stats *stats)
+{
+	bool is_thp = folio_test_pmd_mappable(folio);
+	int rc;
+
+	/* Large folio NUMA faulting doesn't split to retry. */
+	if (!folio_test_large(folio) || reason == MR_NUMA_MISPLACED)
+		return -ENOMEM;
+
+	rc = try_split_folio(folio, split_folios, mode);
+	if (!rc) {
+		stats->nr_thp_split += is_thp;
+		stats->nr_split++;
+		return 0;
+	}
+
+	/*
+	 * Try again to split large folio to mitigate the failure of longterm
+	 * pinning.
+	 */
+	if (reason == MR_LONGTERM_PIN && rc == -EAGAIN)
+		return -EAGAIN;
+
+	return -ENOMEM;
+}
+
 /*
  * Unmap the folios in @from, queuing successfully unmapped sources on
  * @unmap_folios and their destination on @dst_folios.  Increment @nr_failed for
@@ -1829,10 +1857,8 @@ static int migrate_folios_unmap(struct list_head *from,
 	int nr_retry_pages = 0;
 	int pass = 0;
 	bool is_thp = false;
-	bool is_large = false;
 	struct folio *folio, *folio2, *dst = NULL;
-	int rc, nr_pages;
-	bool nosplit = (reason == MR_NUMA_MISPLACED);
+	int rc, split_rc, nr_pages;
 
 	for (pass = 0; pass < nr_pass && retry; pass++) {
 		retry = 0;
@@ -1840,7 +1866,6 @@ static int migrate_folios_unmap(struct list_head *from,
 		nr_retry_pages = 0;
 
 		list_for_each_entry_safe(folio, folio2, from, lru) {
-			is_large = folio_test_large(folio);
 			is_thp = folio_test_pmd_mappable(folio);
 			nr_pages = folio_nr_pages(folio);
 
@@ -1937,28 +1962,20 @@ static int migrate_folios_unmap(struct list_head *from,
 				 */
 				*nr_failed += 1;
 				stats->nr_thp_failed += is_thp;
-				/* Large folio NUMA faulting doesn't split to retry. */
-				if (is_large && !nosplit) {
-					int ret = try_split_folio(folio, split_folios, mode);
-
-					if (!ret) {
-						stats->nr_thp_split += is_thp;
-						stats->nr_split++;
-						break;
-					} else if (reason == MR_LONGTERM_PIN &&
-						   ret == -EAGAIN) {
-						/*
-						 * Try again to split large folio to
-						 * mitigate the failure of longterm pinning.
-						 */
-						retry++;
-						thp_retry += is_thp;
-						nr_retry_pages += nr_pages;
-						/* Undo duplicated failure counting. */
-						*nr_failed -= 1;
-						stats->nr_thp_failed -= is_thp;
-						break;
-					}
+
+				split_rc = migrate_folio_split_on_alloc_fail(folio,
+						split_folios, mode, reason, stats);
+				if (!split_rc)
+					break;
+				/* Retry this folio in a later pass. */
+				if (split_rc == -EAGAIN) {
+					retry++;
+					thp_retry += is_thp;
+					nr_retry_pages += nr_pages;
+					/* Undo duplicated failure counting. */
+					*nr_failed -= 1;
+					stats->nr_thp_failed -= is_thp;
+					break;
 				}
 
 				stats->nr_failed_pages += nr_pages + nr_retry_pages;

-- 
2.43.0

[PATCH RFC 04/11] mm/migrate: use a dedicated list for hugetlb folios

[Shivank Garg]

migrate_hugetlbs() scans the full source list on every retry pass, even
though it only handles hugetlb folios.

Move hugetlb folios to a local list before migration and move any remaining
folios to @ret_folios. This avoids scanning unrelated folios and removes
the later hugetlb check from migrate_pages().

No functional change intended.

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 mm/migrate.c | 43 ++++++++++++++++++++++---------------------
 1 file changed, 22 insertions(+), 21 deletions(-)

diff --git a/mm/migrate.c b/mm/migrate.c
index 510dfffa7235..7a136cec275f 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -1629,11 +1629,12 @@ struct migrate_pages_stats {
 };
 
 /*
- * Returns the number of hugetlb folios that were not migrated, or an error code
- * after NR_MAX_MIGRATE_PAGES_RETRY attempts or if no hugetlb folios are movable
- * any more because the list has become empty or no retryable hugetlb folios
- * exist any more. It is caller's responsibility to call putback_movable_pages()
- * only if ret != 0.
+ * Move hugetlb folios from @from to a local list and try to migrate each folio
+ * up to NR_MAX_MIGRATE_PAGES_RETRY times. Any remaining folios are moved to
+ * @ret_folios list.
+ *
+ * Return the number of failed folios, or a negative errno.
+ *
  */
 static int migrate_hugetlbs(struct list_head *from, new_folio_t get_new_folio,
 			    free_folio_t put_new_folio, unsigned long private,
@@ -1646,16 +1647,18 @@ static int migrate_hugetlbs(struct list_head *from, new_folio_t get_new_folio,
 	int nr_retry_pages = 0;
 	int pass = 0;
 	struct folio *folio, *folio2;
-	int rc, nr_pages;
+	int rc, ret, nr_pages;
+	LIST_HEAD(hugetlbs);
+
+	list_for_each_entry_safe(folio, folio2, from, lru)
+		if (folio_test_hugetlb(folio))
+			list_move_tail(&folio->lru, &hugetlbs);
 
 	for (pass = 0; pass < NR_MAX_MIGRATE_PAGES_RETRY && retry; pass++) {
 		retry = 0;
 		nr_retry_pages = 0;
 
-		list_for_each_entry_safe(folio, folio2, from, lru) {
-			if (!folio_test_hugetlb(folio))
-				continue;
-
+		list_for_each_entry_safe(folio, folio2, &hugetlbs, lru) {
 			nr_pages = folio_nr_pages(folio);
 
 			cond_resched();
@@ -1681,18 +1684,19 @@ static int migrate_hugetlbs(struct list_head *from, new_folio_t get_new_folio,
 			/*
 			 * The rules are:
 			 *	0: hugetlb folio will be put back
-			 *	-EAGAIN: stay on the from list
-			 *	-ENOMEM: stay on the from list
+			 *	-EAGAIN: stay on hugetlbs, retried by a later pass
+			 *	-ENOMEM: give up; rest of the list goes to ret_folios
 			 *	Other errno: put on ret_folios list
 			 */
-			switch(rc) {
+			switch (rc) {
 			case -ENOMEM:
 				/*
 				 * When memory is low, don't bother to try to migrate
 				 * other folios, just exit.
 				 */
 				stats->nr_failed_pages += nr_pages + nr_retry_pages;
-				return -ENOMEM;
+				ret = -ENOMEM;
+				goto out;
 			case -EAGAIN:
 				retry++;
 				nr_retry_pages += nr_pages;
@@ -1720,8 +1724,11 @@ static int migrate_hugetlbs(struct list_head *from, new_folio_t get_new_folio,
 	 */
 	nr_failed += retry;
 	stats->nr_failed_pages += nr_retry_pages;
+	ret = nr_failed;
+out:
+	list_splice_tail(&hugetlbs, ret_folios);
 
-	return nr_failed;
+	return ret;
 }
 
 static void migrate_folios_move(struct list_head *src_folios,
@@ -2161,12 +2168,6 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 again:
 	nr_pages = 0;
 	list_for_each_entry_safe(folio, folio2, from, lru) {
-		/* Retried hugetlb folios will be kept in list  */
-		if (folio_test_hugetlb(folio)) {
-			list_move_tail(&folio->lru, &ret_folios);
-			continue;
-		}
-
 		nr_pages += folio_nr_pages(folio);
 		if (nr_pages >= NR_MAX_BATCHED_MIGRATION)
 			break;

-- 
2.43.0

[PATCH RFC 05/11] mm/migrate: add a dedicated movable_ops migration pass

[Shivank Garg]

Pages with movable_ops transfer ownership through a driver callback and do
not need the unmap, TLB flush, copy, or LRU handling used for LRU folios.
These pages are expected to stop being represented as folios. Separating
their migration from the LRU folio path prepares for that conversion.

Add migrate_movable_ops_pages() before LRU folio migration to handle
allocation, locking, retries, and cleanup on a separate list. Return
remaining pages through @ret_folios and remove the now-unreachable
movable_ops branches from the LRU path.

This changes three details of movable_ops migration:
 - Pages are processed before LRU folios rather than in source-list order.
 - Synchronous callers use the requested mode from the first attempt
   instead of the LRU asynchronous pre-pass.
 - A callback returning -EAGAIN ends the current attempt, so a later
   retry allocates a fresh destination.
 - MIGRATE_SYNC_LIGHT waits for the page lock instead of giving up,
   because !uptodate check is not applicable for movable_ops
   pages.

Suggested-by: Zi Yan <ziy@nvidia.com>
Suggested-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 mm/migrate.c | 156 +++++++++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 140 insertions(+), 16 deletions(-)

diff --git a/mm/migrate.c b/mm/migrate.c
index 7a136cec275f..55f352efdf2b 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -1200,7 +1200,7 @@ static void migrate_folio_undo_dst(struct folio *dst, bool locked,
 static void migrate_folio_done(struct folio *src,
 			       enum migrate_reason reason)
 {
-	if (likely(!page_has_movable_ops(&src->page)) && reason != MR_DEMOTION)
+	if (reason != MR_DEMOTION)
 		mod_node_page_state(folio_pgdat(src), NR_ISOLATED_ANON +
 				    folio_is_file_lru(src), -folio_nr_pages(src));
 
@@ -1309,11 +1309,6 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,
 		goto out;
 	dst_locked = true;
 
-	if (unlikely(page_has_movable_ops(&src->page))) {
-		__migrate_folio_record(dst, old_folio_state, anon_vma);
-		return 0;
-	}
-
 	/*
 	 * Corner case handling:
 	 * 1. When a new swap-cache page is read into, it is added to the LRU
@@ -1376,13 +1371,6 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
 	prev = dst->lru.prev;
 	list_del(&dst->lru);
 
-	if (unlikely(page_has_movable_ops(&src->page))) {
-		rc = migrate_movable_ops_page(&dst->page, &src->page, mode);
-		if (rc)
-			goto out;
-		goto out_unlock_both;
-	}
-
 	if (folio_order(src) > 1 &&
 	    !data_race(list_empty(&src->_deferred_list))) {
 		src_deferred_split = true;
@@ -1418,7 +1406,6 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
 	if (old_folio_state & FOLIO_WAS_MAPPED)
 		remove_migration_ptes(src, dst, 0);
 
-out_unlock_both:
 	folio_unlock(dst);
 	folio_set_owner_migrate_reason(dst, reason);
 	/*
@@ -1816,6 +1803,135 @@ static void migrate_folios_undo(struct list_head *src_folios,
 	}
 }
 
+/*
+ * Migrate one isolated movable_ops page. Keep @src on its list for retry and
+ * move it to @ret_folios on permanent failure.
+ *
+ * Return 0 on success or a negative error.
+ */
+static int move_movable_ops_page(struct folio *src,
+		new_folio_t get_new_folio, free_folio_t put_new_folio,
+		unsigned long private, enum migrate_mode mode,
+		enum migrate_reason reason, struct list_head *ret_folios)
+{
+	struct folio *dst;
+	int rc = -EAGAIN;
+
+	dst = get_new_folio(src, private);
+	if (!dst)
+		return -ENOMEM;
+
+	if (!folio_trylock(src)) {
+		if (mode == MIGRATE_ASYNC)
+			goto out_put_dst;
+		if (current->flags & PF_MEMALLOC)
+			goto out_put_dst;
+		folio_lock(src);
+	}
+
+	if (unlikely(!folio_trylock(dst)))
+		goto out_unlock_src;
+
+	rc = migrate_movable_ops_page(&dst->page, &src->page, mode);
+	folio_unlock(dst);
+	if (rc)
+		goto out_unlock_src;
+
+	folio_set_owner_migrate_reason(dst, reason);
+	/* Drop migration's reference after transferring ownership to dst. */
+	folio_put(dst);
+
+	list_del(&src->lru);
+	folio_unlock(src);
+
+	if (reason != MR_MEMORY_FAILURE)
+		folio_put(src);
+
+	return 0;
+
+out_unlock_src:
+	folio_unlock(src);
+out_put_dst:
+	if (put_new_folio)
+		put_new_folio(dst, private);
+	else
+		folio_put(dst);
+
+	if (rc != -EAGAIN)
+		list_move_tail(&src->lru, ret_folios);
+
+	return rc;
+}
+
+/*
+ * Move movable_ops pages from @from to a local list and try to migrate each
+ * page up to NR_MAX_MIGRATE_PAGES_RETRY times. Any remaining pages are moved
+ * to @ret_folios.
+ *
+ * Return the number of failed pages, or a negative error.
+ */
+static int migrate_movable_ops_pages(struct list_head *from,
+		new_folio_t get_new_folio, free_folio_t put_new_folio,
+		unsigned long private, enum migrate_mode mode,
+		enum migrate_reason reason, struct migrate_pages_stats *stats,
+		struct list_head *ret_folios)
+{
+	int retry = 1;
+	int nr_failed = 0;
+	int pass;
+	struct folio *folio, *folio2;
+	int rc, ret;
+	LIST_HEAD(movable_ops_pages);
+
+	list_for_each_entry_safe(folio, folio2, from, lru)
+		if (page_has_movable_ops(&folio->page))
+			list_move_tail(&folio->lru, &movable_ops_pages);
+
+	for (pass = 0; pass < NR_MAX_MIGRATE_PAGES_RETRY && retry; pass++) {
+		retry = 0;
+
+		list_for_each_entry_safe(folio, folio2, &movable_ops_pages, lru) {
+			cond_resched();
+
+			rc = move_movable_ops_page(folio, get_new_folio,
+						   put_new_folio, private,
+						   mode, reason, ret_folios);
+			switch (rc) {
+			case -ENOMEM:
+				/* Count this page and those awaiting retry. */
+				nr_failed += 1 + retry;
+				ret = -ENOMEM;
+				goto out;
+			case -EAGAIN:
+				retry++;
+				break;
+			case 0:
+				stats->nr_succeeded++;
+				break;
+			default:
+				nr_failed++;
+				break;
+			}
+		}
+	}
+	/* Count pages that exhausted the retry limit. */
+	nr_failed += retry;
+	ret = nr_failed;
+out:
+	/* movable_ops pages are order-0, so one failed page each. */
+	stats->nr_failed_pages += nr_failed;
+	list_splice_tail(&movable_ops_pages, ret_folios);
+
+	return ret;
+}
+
+/*
+ * Split a large folio after destination allocation fails and queue the resulting
+ * folios on @split_folios.
+ *
+ * Return: 0 on success, -EAGAIN to retry splitting later, or -ENOMEM to stop the
+ * unmap phase.
+ */
 static int migrate_folio_split_on_alloc_fail(struct folio *folio,
 		struct list_head *split_folios, enum migrate_mode mode,
 		enum migrate_reason reason, struct migrate_pages_stats *stats)
@@ -1940,8 +2056,7 @@ static int migrate_folios_unmap(struct list_head *from,
 			 * If we are holding the last folio reference, the folio
 			 * was freed from under us, so just drop our reference.
 			 */
-			if (likely(!page_has_movable_ops(&folio->page)) &&
-			    folio_ref_count(folio) == 1) {
+			if (folio_ref_count(folio) == 1) {
 				folio_clear_active(folio);
 				folio_clear_unevictable(folio);
 				list_del(&folio->lru);
@@ -2165,6 +2280,15 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 	if (rc_gather < 0)
 		goto out;
 
+	rc = migrate_movable_ops_pages(from, get_new_folio, put_new_folio,
+				       private, mode, reason, &stats,
+				       &ret_folios);
+	if (rc < 0) {
+		rc_gather = rc;
+		goto out;
+	}
+	rc_gather += rc;
+
 again:
 	nr_pages = 0;
 	list_for_each_entry_safe(folio, folio2, from, lru) {

-- 
2.43.0

[PATCH RFC 06/11] mm/migrate: rename migrate_pages_batch() to migrate_folios_batch()

[Shivank Garg]

migrate_pages_batch() operates on folios and calls the folio-based unmap,
move and undo helpers. Now that movable_ops-based page migration has been
separated out, rename it to migrate_folios_batch().

No functional changes intended.

Suggested-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 mm/migrate.c | 22 +++++++++++-----------
 1 file changed, 11 insertions(+), 11 deletions(-)

diff --git a/mm/migrate.c b/mm/migrate.c
index 55f352efdf2b..163f13344fd2 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -2003,7 +2003,7 @@ static int migrate_folios_unmap(struct list_head *from,
 			 * always reports success when its fromlist is empty.
 			 * stats->nr_thp_failed should be increased too,
 			 * otherwise stats inconsistency will happen when
-			 * migrate_pages_batch is called via migrate_pages()
+			 * migrate_folios_batch is called via migrate_pages()
 			 * with MIGRATE_SYNC and MIGRATE_ASYNC.
 			 *
 			 * Only check it without removing it from the list.
@@ -2135,7 +2135,7 @@ static int migrate_folios_unmap(struct list_head *from,
 }
 
 /*
- * migrate_pages_batch() first unmaps as many folios in the source list as
+ * migrate_folios_batch() first unmaps as many folios in the source list as
  * possible, flushes the TLBs, then moves the unmapped folios.
  *
  * Only MIGRATE_ASYNC may batch multiple folios.  Waiting for a lock or bit
@@ -2143,7 +2143,7 @@ static int migrate_folios_unmap(struct list_head *from,
  * Therefore, if mode != MIGRATE_ASYNC, the source list must contain at most
  * one folio.
  */
-static int migrate_pages_batch(struct list_head *from,
+static int migrate_folios_batch(struct list_head *from,
 		new_folio_t get_new_folio, free_folio_t put_new_folio,
 		unsigned long private, enum migrate_mode mode, enum migrate_reason reason,
 		struct list_head *ret_folios, struct list_head *split_folios,
@@ -2194,9 +2194,9 @@ static int migrate_pages_sync(struct list_head *from, new_folio_t get_new_folio,
 
 	memset(&astats, 0, sizeof(astats));
 	/* Try to migrate in batch with MIGRATE_ASYNC mode firstly */
-	rc = migrate_pages_batch(from, get_new_folio, put_new_folio, private, MIGRATE_ASYNC,
-				 reason, &folios, split_folios, &astats,
-				 NR_MAX_MIGRATE_ASYNC_RETRY);
+	rc = migrate_folios_batch(from, get_new_folio, put_new_folio, private,
+			MIGRATE_ASYNC, reason, &folios, split_folios, &astats,
+			NR_MAX_MIGRATE_ASYNC_RETRY);
 	stats->nr_succeeded += astats.nr_succeeded;
 	stats->nr_thp_succeeded += astats.nr_thp_succeeded;
 	stats->nr_thp_split += astats.nr_thp_split;
@@ -2221,9 +2221,9 @@ static int migrate_pages_sync(struct list_head *from, new_folio_t get_new_folio,
 	list_splice_tail_init(&folios, from);
 	while (!list_empty(from)) {
 		list_move(from->next, &folios);
-		rc = migrate_pages_batch(&folios, get_new_folio, put_new_folio,
-					 private, mode, reason, ret_folios,
-					 split_folios, stats, NR_MAX_MIGRATE_SYNC_RETRY);
+		rc = migrate_folios_batch(&folios, get_new_folio, put_new_folio,
+				private, mode, reason, ret_folios, split_folios,
+				stats, NR_MAX_MIGRATE_SYNC_RETRY);
 		list_splice_tail_init(&folios, ret_folios);
 		if (rc < 0)
 			return rc;
@@ -2301,7 +2301,7 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 	else
 		list_splice_init(from, &folios);
 	if (mode == MIGRATE_ASYNC)
-		rc = migrate_pages_batch(&folios, get_new_folio, put_new_folio,
+		rc = migrate_folios_batch(&folios, get_new_folio, put_new_folio,
 				private, mode, reason, &ret_folios,
 				&split_folios, &stats,
 				NR_MAX_MIGRATE_PAGES_RETRY);
@@ -2321,7 +2321,7 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 		 * is counted as 1 failure already.  And, we only try to migrate
 		 * with minimal effort, force MIGRATE_ASYNC mode and retry once.
 		 */
-		migrate_pages_batch(&split_folios, get_new_folio,
+		migrate_folios_batch(&split_folios, get_new_folio,
 				put_new_folio, private, MIGRATE_ASYNC, reason,
 				&ret_folios, NULL, &stats, 1);
 		list_splice_tail_init(&split_folios, &ret_folios);

-- 
2.43.0

[PATCH RFC 07/11] mm/migrate: add migrate_lru_folios() entry point

[Shivank Garg]

migrate_pages() currently handles the asynchronous and synchronous LRU
paths separately and retries split folios itself.

Move this logic into migrate_lru_folios(), leaving migrate_pages() to
dispatch hugetlb, movable_ops, and LRU folios through their respective
entry points.

Keep the existing retry and accounting behavior unchanged.

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 mm/migrate.c | 70 ++++++++++++++++++++++++++++++++++--------------------------
 1 file changed, 40 insertions(+), 30 deletions(-)

diff --git a/mm/migrate.c b/mm/migrate.c
index 163f13344fd2..d39149cbd43d 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -2182,20 +2182,34 @@ static int migrate_folios_batch(struct list_head *from,
 	return rc;
 }
 
-static int migrate_pages_sync(struct list_head *from, new_folio_t get_new_folio,
+/*
+ * Migrate LRU folios.  MIGRATE_ASYNC processes a batch directly.  The
+ * other modes first make a short asynchronous pass, then retry each remaining
+ * folio separately in the requested mode.
+ */
+static int migrate_lru_folios(struct list_head *from, new_folio_t get_new_folio,
 		free_folio_t put_new_folio, unsigned long private,
 		enum migrate_mode mode, enum migrate_reason reason,
-		struct list_head *ret_folios, struct list_head *split_folios,
+		struct list_head *ret_folios,
 		struct migrate_pages_stats *stats)
 {
 	int rc, nr_failed = 0;
 	LIST_HEAD(folios);
+	LIST_HEAD(split_folios);
 	struct migrate_pages_stats astats;
 
+	if (mode == MIGRATE_ASYNC) {
+		rc = migrate_folios_batch(from, get_new_folio, put_new_folio,
+				private, mode, reason, ret_folios,
+				&split_folios, stats,
+				NR_MAX_MIGRATE_PAGES_RETRY);
+		goto out;
+	}
+
 	memset(&astats, 0, sizeof(astats));
 	/* Try to migrate in batch with MIGRATE_ASYNC mode firstly */
 	rc = migrate_folios_batch(from, get_new_folio, put_new_folio, private,
-			MIGRATE_ASYNC, reason, &folios, split_folios, &astats,
+			MIGRATE_ASYNC, reason, &folios, &split_folios, &astats,
 			NR_MAX_MIGRATE_ASYNC_RETRY);
 	stats->nr_succeeded += astats.nr_succeeded;
 	stats->nr_thp_succeeded += astats.nr_thp_succeeded;
@@ -2205,7 +2219,7 @@ static int migrate_pages_sync(struct list_head *from, new_folio_t get_new_folio,
 		stats->nr_failed_pages += astats.nr_failed_pages;
 		stats->nr_thp_failed += astats.nr_thp_failed;
 		list_splice_tail(&folios, ret_folios);
-		return rc;
+		goto out;
 	}
 	stats->nr_thp_failed += astats.nr_thp_split;
 	/*
@@ -2222,15 +2236,31 @@ static int migrate_pages_sync(struct list_head *from, new_folio_t get_new_folio,
 	while (!list_empty(from)) {
 		list_move(from->next, &folios);
 		rc = migrate_folios_batch(&folios, get_new_folio, put_new_folio,
-				private, mode, reason, ret_folios, split_folios,
-				stats, NR_MAX_MIGRATE_SYNC_RETRY);
+				private, mode, reason, ret_folios,
+				&split_folios, stats,
+				NR_MAX_MIGRATE_SYNC_RETRY);
 		list_splice_tail_init(&folios, ret_folios);
 		if (rc < 0)
-			return rc;
+			goto out;
 		nr_failed += rc;
 	}
+	rc = nr_failed;
+out:
+	if (rc < 0) {
+		list_splice_tail(&split_folios, ret_folios);
+	} else if (!list_empty(&split_folios)) {
+		/*
+		 * Folios split along the way get one asynchronous attempt at
+		 * their new order.  Their failure is not counted: the large
+		 * folio they came from was already counted as one failure.
+		 */
+		migrate_folios_batch(&split_folios, get_new_folio, put_new_folio,
+				private, MIGRATE_ASYNC, reason, ret_folios,
+				NULL, stats, 1);
+		list_splice_tail_init(&split_folios, ret_folios);
+	}
 
-	return nr_failed;
+	return rc;
 }
 
 /*
@@ -2268,7 +2298,6 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 	struct folio *folio, *folio2;
 	LIST_HEAD(folios);
 	LIST_HEAD(ret_folios);
-	LIST_HEAD(split_folios);
 	struct migrate_pages_stats stats;
 
 	trace_mm_migrate_pages_start(mode, reason);
@@ -2300,32 +2329,13 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 		list_cut_before(&folios, from, &folio2->lru);
 	else
 		list_splice_init(from, &folios);
-	if (mode == MIGRATE_ASYNC)
-		rc = migrate_folios_batch(&folios, get_new_folio, put_new_folio,
-				private, mode, reason, &ret_folios,
-				&split_folios, &stats,
-				NR_MAX_MIGRATE_PAGES_RETRY);
-	else
-		rc = migrate_pages_sync(&folios, get_new_folio, put_new_folio,
-				private, mode, reason, &ret_folios,
-				&split_folios, &stats);
+	rc = migrate_lru_folios(&folios, get_new_folio, put_new_folio,
+			private, mode, reason, &ret_folios, &stats);
 	list_splice_tail_init(&folios, &ret_folios);
 	if (rc < 0) {
 		rc_gather = rc;
-		list_splice_tail(&split_folios, &ret_folios);
 		goto out;
 	}
-	if (!list_empty(&split_folios)) {
-		/*
-		 * Failure isn't counted since all split folios of a large folio
-		 * is counted as 1 failure already.  And, we only try to migrate
-		 * with minimal effort, force MIGRATE_ASYNC mode and retry once.
-		 */
-		migrate_folios_batch(&split_folios, get_new_folio,
-				put_new_folio, private, MIGRATE_ASYNC, reason,
-				&ret_folios, NULL, &stats, 1);
-		list_splice_tail_init(&split_folios, &ret_folios);
-	}
 	rc_gather += rc;
 	if (!list_empty(from))
 		goto again;

-- 
2.43.0

[PATCH RFC 08/11] mm/migrate: move LRU batching into migrate_lru_folios()

[Shivank Garg]

The NR_MAX_BATCHED_MIGRATION limit is specific to LRU folio migration,
where the batch engine can keep multiple folios locked at once.

Move the batching loop into migrate_lru_folios() and rename the per-batch
worker to __migrate_lru_folios(). This leaves migrate_pages() responsible
only for dispatching the different migration classes.

No functional change intended.

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 mm/migrate.c | 62 ++++++++++++++++++++++++++++++++++++++++--------------------
 1 file changed, 42 insertions(+), 20 deletions(-)

diff --git a/mm/migrate.c b/mm/migrate.c
index d39149cbd43d..8914358cc434 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -2183,11 +2183,11 @@ static int migrate_folios_batch(struct list_head *from,
 }
 
 /*
- * Migrate LRU folios.  MIGRATE_ASYNC processes a batch directly.  The
+ * Migrate one batch of LRU folios.  MIGRATE_ASYNC processes it directly.  The
  * other modes first make a short asynchronous pass, then retry each remaining
  * folio separately in the requested mode.
  */
-static int migrate_lru_folios(struct list_head *from, new_folio_t get_new_folio,
+static int __migrate_lru_folios(struct list_head *from, new_folio_t get_new_folio,
 		free_folio_t put_new_folio, unsigned long private,
 		enum migrate_mode mode, enum migrate_reason reason,
 		struct list_head *ret_folios,
@@ -2263,6 +2263,45 @@ static int migrate_lru_folios(struct list_head *from, new_folio_t get_new_folio,
 	return rc;
 }
 
+/*
+ * Migrate LRU folios.  The batch engine keeps every folio of a batch locked
+ * at once, so feed it at most NR_MAX_BATCHED_MIGRATION pages at a time
+ * rather than however many the caller supplied.
+ */
+static int migrate_lru_folios(struct list_head *from, new_folio_t get_new_folio,
+		free_folio_t put_new_folio, unsigned long private,
+		enum migrate_mode mode, enum migrate_reason reason,
+		struct list_head *ret_folios,
+		struct migrate_pages_stats *stats)
+{
+	struct folio *folio, *folio2;
+	int rc, nr_failed = 0;
+	int nr_pages;
+	LIST_HEAD(folios);
+
+	while (!list_empty(from)) {
+		nr_pages = 0;
+		list_for_each_entry_safe(folio, folio2, from, lru) {
+			nr_pages += folio_nr_pages(folio);
+			if (nr_pages >= NR_MAX_BATCHED_MIGRATION)
+				break;
+		}
+		if (nr_pages >= NR_MAX_BATCHED_MIGRATION)
+			list_cut_before(&folios, from, &folio2->lru);
+		else
+			list_splice_init(from, &folios);
+
+		rc = __migrate_lru_folios(&folios, get_new_folio, put_new_folio,
+				private, mode, reason, ret_folios, stats);
+		list_splice_tail_init(&folios, ret_folios);
+		if (rc < 0)
+			return rc;
+		nr_failed += rc;
+	}
+
+	return nr_failed;
+}
+
 /*
  * migrate_pages - migrate the folios specified in a list, to the free folios
  *		   supplied as the target for the page migration
@@ -2294,9 +2333,6 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 		enum migrate_mode mode, enum migrate_reason reason, unsigned int *ret_succeeded)
 {
 	int rc, rc_gather;
-	int nr_pages;
-	struct folio *folio, *folio2;
-	LIST_HEAD(folios);
 	LIST_HEAD(ret_folios);
 	struct migrate_pages_stats stats;
 
@@ -2318,27 +2354,13 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 	}
 	rc_gather += rc;
 
-again:
-	nr_pages = 0;
-	list_for_each_entry_safe(folio, folio2, from, lru) {
-		nr_pages += folio_nr_pages(folio);
-		if (nr_pages >= NR_MAX_BATCHED_MIGRATION)
-			break;
-	}
-	if (nr_pages >= NR_MAX_BATCHED_MIGRATION)
-		list_cut_before(&folios, from, &folio2->lru);
-	else
-		list_splice_init(from, &folios);
-	rc = migrate_lru_folios(&folios, get_new_folio, put_new_folio,
+	rc = migrate_lru_folios(from, get_new_folio, put_new_folio,
 			private, mode, reason, &ret_folios, &stats);
-	list_splice_tail_init(&folios, &ret_folios);
 	if (rc < 0) {
 		rc_gather = rc;
 		goto out;
 	}
 	rc_gather += rc;
-	if (!list_empty(from))
-		goto again;
 out:
 	/*
 	 * Put the permanent failure folio back to migration list, they

-- 
2.43.0

[PATCH RFC 09/11] mm/migrate: thread migration policy through a control struct

[Shivank Garg]

Migration mode and reason describe one migration invocation, but are passed
separately through every migration engine and phase. Adding more policy
would require another parameter through the same call graph or overloading
migrate_mode with unrelated information.

Add struct migrate_control, initialize it on the migrate_pages() stack, and
pass it as const through the internal migration helpers. This provides one
place for additional policy, such as copy-cache hints.

Use a derived control for asynchronous pre-passes and split-folio retries.
Keep the public migration and filesystem callback interfaces unchanged in
this patch and will be handled separately in upcoming patch.

No functional change intended.

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 include/linux/migrate_mode.h |  14 +++++
 mm/migrate.c                 | 144 ++++++++++++++++++++++---------------------
 2 files changed, 88 insertions(+), 70 deletions(-)

diff --git a/include/linux/migrate_mode.h b/include/linux/migrate_mode.h
index 05102d4d2490..388acddfdae8 100644
--- a/include/linux/migrate_mode.h
+++ b/include/linux/migrate_mode.h
@@ -29,4 +29,18 @@ enum migrate_reason {
 	MR_TYPES
 };
 
+/**
+ * struct migrate_control - Policy for one migration invocation
+ * @mode: Blocking discipline for migration operations
+ * @reason: Reason the migration was requested
+ *
+ * Per-folio completion state does not belong here.  Callers keep this
+ * structure on the stack; migration must not allocate it under memory
+ * pressure.
+ */
+struct migrate_control {
+	enum migrate_mode mode;
+	enum migrate_reason reason;
+};
+
 #endif		/* MIGRATE_MODE_H_INCLUDED */
diff --git a/mm/migrate.c b/mm/migrate.c
index 8914358cc434..e4bc7122076e 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -1103,7 +1103,7 @@ static int fallback_migrate_folio(struct address_space *mapping,
  *     0 - success
  */
 static int move_to_new_folio(struct folio *dst, struct folio *src,
-				enum migrate_mode mode)
+		const struct migrate_control *ctl)
 {
 	struct address_space *mapping = folio_mapping(src);
 	int rc = -EAGAIN;
@@ -1112,7 +1112,7 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,
 	VM_BUG_ON_FOLIO(!folio_test_locked(dst), dst);
 
 	if (!mapping)
-		rc = migrate_folio(mapping, dst, src, mode);
+		rc = migrate_folio(mapping, dst, src, ctl->mode);
 	else if (mapping_inaccessible(mapping))
 		rc = -EOPNOTSUPP;
 	else if (mapping->a_ops->migrate_folio)
@@ -1123,10 +1123,9 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,
 		 * migrate_folio callback. This is the most common path
 		 * for page migration.
 		 */
-		rc = mapping->a_ops->migrate_folio(mapping, dst, src,
-							mode);
+		rc = mapping->a_ops->migrate_folio(mapping, dst, src, ctl->mode);
 	else
-		rc = fallback_migrate_folio(mapping, dst, src, mode);
+		rc = fallback_migrate_folio(mapping, dst, src, ctl->mode);
 
 	if (!rc) {
 		/*
@@ -1198,13 +1197,13 @@ static void migrate_folio_undo_dst(struct folio *dst, bool locked,
 
 /* Cleanup src folio upon migration success */
 static void migrate_folio_done(struct folio *src,
-			       enum migrate_reason reason)
+		const struct migrate_control *ctl)
 {
-	if (reason != MR_DEMOTION)
+	if (ctl->reason != MR_DEMOTION)
 		mod_node_page_state(folio_pgdat(src), NR_ISOLATED_ANON +
 				    folio_is_file_lru(src), -folio_nr_pages(src));
 
-	if (reason != MR_MEMORY_FAILURE)
+	if (ctl->reason != MR_MEMORY_FAILURE)
 		/* We release the page in page_handle_poison. */
 		folio_put(src);
 }
@@ -1212,8 +1211,8 @@ static void migrate_folio_done(struct folio *src,
 /* Obtain the lock on page, remove all ptes. */
 static int migrate_folio_unmap(new_folio_t get_new_folio,
 		free_folio_t put_new_folio, unsigned long private,
-		struct folio *src, struct folio **dstp, enum migrate_mode mode,
-		struct list_head *ret)
+		struct folio *src, struct folio **dstp,
+		const struct migrate_control *ctl, struct list_head *ret)
 {
 	struct folio *dst;
 	int rc = -EAGAIN;
@@ -1230,7 +1229,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,
 	dst->migrate_info = 0;
 
 	if (!folio_trylock(src)) {
-		if (mode == MIGRATE_ASYNC)
+		if (ctl->mode == MIGRATE_ASYNC)
 			goto out;
 
 		/*
@@ -1254,7 +1253,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,
 		 * inserting a page into the page table), but it's not
 		 * worth waiting for I/O.
 		 */
-		if (mode == MIGRATE_SYNC_LIGHT && !folio_test_uptodate(src))
+		if (ctl->mode == MIGRATE_SYNC_LIGHT && !folio_test_uptodate(src))
 			goto out;
 
 		folio_lock(src);
@@ -1270,7 +1269,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,
 		 * the retry loop is too short and in the sync-light case,
 		 * the overhead of stalling is too much
 		 */
-		switch (mode) {
+		switch (ctl->mode) {
 		case MIGRATE_SYNC:
 			break;
 		default:
@@ -1330,7 +1329,8 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,
 		/* Establish migration ptes */
 		VM_BUG_ON_FOLIO(folio_test_anon(src) &&
 			       !folio_test_ksm(src) && !anon_vma, src);
-		try_to_migrate(src, mode == MIGRATE_ASYNC ? TTU_BATCH_FLUSH : 0);
+		try_to_migrate(src, ctl->mode == MIGRATE_ASYNC ?
+			TTU_BATCH_FLUSH : 0);
 		old_folio_state |= FOLIO_WAS_MAPPED;
 	}
 
@@ -1356,9 +1356,8 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,
 
 /* Migrate the folio to the newly allocated folio in dst. */
 static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
-			      struct folio *src, struct folio *dst,
-			      enum migrate_mode mode, enum migrate_reason reason,
-			      struct list_head *ret)
+		struct folio *src, struct folio *dst,
+		const struct migrate_control *ctl, struct list_head *ret)
 {
 	int rc;
 	int old_folio_state = 0;
@@ -1377,7 +1376,7 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
 		src_partially_mapped = folio_test_partially_mapped(src);
 	}
 
-	rc = move_to_new_folio(dst, src, mode);
+	rc = move_to_new_folio(dst, src, ctl);
 	if (rc)
 		goto out;
 
@@ -1407,7 +1406,7 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
 		remove_migration_ptes(src, dst, 0);
 
 	folio_unlock(dst);
-	folio_set_owner_migrate_reason(dst, reason);
+	folio_set_owner_migrate_reason(dst, ctl->reason);
 	/*
 	 * If migration is successful, decrease refcount of dst,
 	 * which will not free the page because new page owner increased
@@ -1424,7 +1423,7 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
 	if (anon_vma)
 		put_anon_vma(anon_vma);
 	folio_unlock(src);
-	migrate_folio_done(src, reason);
+	migrate_folio_done(src, ctl);
 
 	return rc;
 out:
@@ -1466,8 +1465,8 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
  */
 static int unmap_and_move_hugetlb_folio(new_folio_t get_new_folio,
 		free_folio_t put_new_folio, unsigned long private,
-		struct folio *src, int force, enum migrate_mode mode,
-		enum migrate_reason reason, struct list_head *ret)
+		struct folio *src, int force, const struct migrate_control *ctl,
+		struct list_head *ret)
 {
 	struct folio *dst;
 	int rc = -EAGAIN;
@@ -1489,7 +1488,7 @@ static int unmap_and_move_hugetlb_folio(new_folio_t get_new_folio,
 	if (!folio_trylock(src)) {
 		if (!force)
 			goto out;
-		switch (mode) {
+		switch (ctl->mode) {
 		case MIGRATE_SYNC:
 			break;
 		default:
@@ -1534,7 +1533,7 @@ static int unmap_and_move_hugetlb_folio(new_folio_t get_new_folio,
 	}
 
 	if (!folio_mapped(src))
-		rc = move_to_new_folio(dst, src, mode);
+		rc = move_to_new_folio(dst, src, ctl);
 
 	if (was_mapped)
 		remove_migration_ptes(src, !rc ? dst : src, ttu);
@@ -1550,7 +1549,7 @@ static int unmap_and_move_hugetlb_folio(new_folio_t get_new_folio,
 		put_anon_vma(anon_vma);
 
 	if (!rc) {
-		move_hugetlb_state(src, dst, reason);
+		move_hugetlb_state(src, dst, ctl->reason);
 		put_new_folio = NULL;
 	}
 
@@ -1625,7 +1624,7 @@ struct migrate_pages_stats {
  */
 static int migrate_hugetlbs(struct list_head *from, new_folio_t get_new_folio,
 			    free_folio_t put_new_folio, unsigned long private,
-			    enum migrate_mode mode, enum migrate_reason reason,
+			    const struct migrate_control *ctl,
 			    struct migrate_pages_stats *stats,
 			    struct list_head *ret_folios)
 {
@@ -1666,8 +1665,8 @@ static int migrate_hugetlbs(struct list_head *from, new_folio_t get_new_folio,
 
 			rc = unmap_and_move_hugetlb_folio(get_new_folio,
 							  put_new_folio, private,
-							  folio, pass > 2, mode,
-							  reason, ret_folios);
+							  folio, pass > 2, ctl,
+							  ret_folios);
 			/*
 			 * The rules are:
 			 *	0: hugetlb folio will be put back
@@ -1721,7 +1720,7 @@ static int migrate_hugetlbs(struct list_head *from, new_folio_t get_new_folio,
 static void migrate_folios_move(struct list_head *src_folios,
 		struct list_head *dst_folios,
 		free_folio_t put_new_folio, unsigned long private,
-		enum migrate_mode mode, enum migrate_reason reason,
+		const struct migrate_control *ctl,
 		struct list_head *ret_folios,
 		struct migrate_pages_stats *stats, int nr_pass,
 		int *nr_failed)
@@ -1748,7 +1747,7 @@ static void migrate_folios_move(struct list_head *src_folios,
 			cond_resched();
 
 			rc = migrate_folio_move(put_new_folio, private,
-					folio, dst, mode, reason, ret_folios);
+					folio, dst, ctl, ret_folios);
 			/*
 			 * The rules are:
 			 *	0: folio will be freed
@@ -1811,8 +1810,8 @@ static void migrate_folios_undo(struct list_head *src_folios,
  */
 static int move_movable_ops_page(struct folio *src,
 		new_folio_t get_new_folio, free_folio_t put_new_folio,
-		unsigned long private, enum migrate_mode mode,
-		enum migrate_reason reason, struct list_head *ret_folios)
+		unsigned long private, const struct migrate_control *ctl,
+		struct list_head *ret_folios)
 {
 	struct folio *dst;
 	int rc = -EAGAIN;
@@ -1822,7 +1821,7 @@ static int move_movable_ops_page(struct folio *src,
 		return -ENOMEM;
 
 	if (!folio_trylock(src)) {
-		if (mode == MIGRATE_ASYNC)
+		if (ctl->mode == MIGRATE_ASYNC)
 			goto out_put_dst;
 		if (current->flags & PF_MEMALLOC)
 			goto out_put_dst;
@@ -1832,19 +1831,19 @@ static int move_movable_ops_page(struct folio *src,
 	if (unlikely(!folio_trylock(dst)))
 		goto out_unlock_src;
 
-	rc = migrate_movable_ops_page(&dst->page, &src->page, mode);
+	rc = migrate_movable_ops_page(&dst->page, &src->page, ctl->mode);
 	folio_unlock(dst);
 	if (rc)
 		goto out_unlock_src;
 
-	folio_set_owner_migrate_reason(dst, reason);
+	folio_set_owner_migrate_reason(dst, ctl->reason);
 	/* Drop migration's reference after transferring ownership to dst. */
 	folio_put(dst);
 
 	list_del(&src->lru);
 	folio_unlock(src);
 
-	if (reason != MR_MEMORY_FAILURE)
+	if (ctl->reason != MR_MEMORY_FAILURE)
 		folio_put(src);
 
 	return 0;
@@ -1872,8 +1871,8 @@ static int move_movable_ops_page(struct folio *src,
  */
 static int migrate_movable_ops_pages(struct list_head *from,
 		new_folio_t get_new_folio, free_folio_t put_new_folio,
-		unsigned long private, enum migrate_mode mode,
-		enum migrate_reason reason, struct migrate_pages_stats *stats,
+		unsigned long private, const struct migrate_control *ctl,
+		struct migrate_pages_stats *stats,
 		struct list_head *ret_folios)
 {
 	int retry = 1;
@@ -1895,7 +1894,7 @@ static int migrate_movable_ops_pages(struct list_head *from,
 
 			rc = move_movable_ops_page(folio, get_new_folio,
 						   put_new_folio, private,
-						   mode, reason, ret_folios);
+						   ctl, ret_folios);
 			switch (rc) {
 			case -ENOMEM:
 				/* Count this page and those awaiting retry. */
@@ -1933,17 +1932,17 @@ static int migrate_movable_ops_pages(struct list_head *from,
  * unmap phase.
  */
 static int migrate_folio_split_on_alloc_fail(struct folio *folio,
-		struct list_head *split_folios, enum migrate_mode mode,
-		enum migrate_reason reason, struct migrate_pages_stats *stats)
+		struct list_head *split_folios, const struct migrate_control *ctl,
+		struct migrate_pages_stats *stats)
 {
 	bool is_thp = folio_test_pmd_mappable(folio);
 	int rc;
 
 	/* Large folio NUMA faulting doesn't split to retry. */
-	if (!folio_test_large(folio) || reason == MR_NUMA_MISPLACED)
+	if (!folio_test_large(folio) || ctl->reason == MR_NUMA_MISPLACED)
 		return -ENOMEM;
 
-	rc = try_split_folio(folio, split_folios, mode);
+	rc = try_split_folio(folio, split_folios, ctl->mode);
 	if (!rc) {
 		stats->nr_thp_split += is_thp;
 		stats->nr_split++;
@@ -1954,7 +1953,7 @@ static int migrate_folio_split_on_alloc_fail(struct folio *folio,
 	 * Try again to split large folio to mitigate the failure of longterm
 	 * pinning.
 	 */
-	if (reason == MR_LONGTERM_PIN && rc == -EAGAIN)
+	if (ctl->reason == MR_LONGTERM_PIN && rc == -EAGAIN)
 		return -EAGAIN;
 
 	return -ENOMEM;
@@ -1970,7 +1969,7 @@ static int migrate_folio_split_on_alloc_fail(struct folio *folio,
  */
 static int migrate_folios_unmap(struct list_head *from,
 		new_folio_t get_new_folio, free_folio_t put_new_folio,
-		unsigned long private, enum migrate_mode mode, enum migrate_reason reason,
+		unsigned long private, const struct migrate_control *ctl,
 		struct list_head *ret_folios, struct list_head *split_folios,
 		struct list_head *unmap_folios, struct list_head *dst_folios,
 		struct migrate_pages_stats *stats, int nr_pass, int *nr_failed)
@@ -2020,7 +2019,7 @@ static int migrate_folios_unmap(struct list_head *from,
 			if (nr_pages > 2 &&
 			   !list_empty(&folio->_deferred_list) &&
 			   folio_test_partially_mapped(folio)) {
-				if (!try_split_folio(folio, split_folios, mode)) {
+				if (!try_split_folio(folio, split_folios, ctl->mode)) {
 					*nr_failed += 1;
 					stats->nr_thp_failed += is_thp;
 					stats->nr_thp_split += is_thp;
@@ -2042,7 +2041,7 @@ static int migrate_folios_unmap(struct list_head *from,
 			if (!thp_migration_supported() && is_thp) {
 				*nr_failed += 1;
 				stats->nr_thp_failed++;
-				if (!try_split_folio(folio, split_folios, mode)) {
+				if (!try_split_folio(folio, split_folios, ctl->mode)) {
 					stats->nr_thp_split++;
 					stats->nr_split++;
 					continue;
@@ -2060,14 +2059,14 @@ static int migrate_folios_unmap(struct list_head *from,
 				folio_clear_active(folio);
 				folio_clear_unevictable(folio);
 				list_del(&folio->lru);
-				migrate_folio_done(folio, reason);
+				migrate_folio_done(folio, ctl);
 				stats->nr_succeeded += nr_pages;
 				stats->nr_thp_succeeded += is_thp;
 				continue;
 			}
 
 			rc = migrate_folio_unmap(get_new_folio, put_new_folio,
-					private, folio, &dst, mode, ret_folios);
+					private, folio, &dst, ctl, ret_folios);
 			/*
 			 * The rules are:
 			 *	0: folio will be put on unmap_folios list,
@@ -2086,7 +2085,7 @@ static int migrate_folios_unmap(struct list_head *from,
 				stats->nr_thp_failed += is_thp;
 
 				split_rc = migrate_folio_split_on_alloc_fail(folio,
-						split_folios, mode, reason, stats);
+						split_folios, ctl, stats);
 				if (!split_rc)
 					break;
 				/* Retry this folio in a later pass. */
@@ -2145,7 +2144,7 @@ static int migrate_folios_unmap(struct list_head *from,
  */
 static int migrate_folios_batch(struct list_head *from,
 		new_folio_t get_new_folio, free_folio_t put_new_folio,
-		unsigned long private, enum migrate_mode mode, enum migrate_reason reason,
+		unsigned long private, const struct migrate_control *ctl,
 		struct list_head *ret_folios, struct list_head *split_folios,
 		struct migrate_pages_stats *stats, int nr_pass)
 {
@@ -2154,11 +2153,11 @@ static int migrate_folios_batch(struct list_head *from,
 	LIST_HEAD(unmap_folios);
 	LIST_HEAD(dst_folios);
 
-	VM_WARN_ON_ONCE(mode != MIGRATE_ASYNC &&
+	VM_WARN_ON_ONCE(ctl->mode != MIGRATE_ASYNC &&
 			!list_empty(from) && !list_is_singular(from));
 
 	rc_saved = migrate_folios_unmap(from, get_new_folio, put_new_folio,
-			private, mode, reason, ret_folios, split_folios,
+			private, ctl, ret_folios, split_folios,
 			&unmap_folios, &dst_folios, stats, nr_pass,
 			&nr_failed);
 	if (rc_saved && list_empty(&unmap_folios)) {
@@ -2171,7 +2170,7 @@ static int migrate_folios_batch(struct list_head *from,
 
 	/* Move the unmapped folios */
 	migrate_folios_move(&unmap_folios, &dst_folios, put_new_folio, private,
-			mode, reason, ret_folios, stats, nr_pass, &nr_failed);
+			ctl, ret_folios, stats, nr_pass, &nr_failed);
 
 	rc = rc_saved ? : nr_failed;
 out:
@@ -2189,27 +2188,29 @@ static int migrate_folios_batch(struct list_head *from,
  */
 static int __migrate_lru_folios(struct list_head *from, new_folio_t get_new_folio,
 		free_folio_t put_new_folio, unsigned long private,
-		enum migrate_mode mode, enum migrate_reason reason,
-		struct list_head *ret_folios,
+		const struct migrate_control *ctl, struct list_head *ret_folios,
 		struct migrate_pages_stats *stats)
 {
 	int rc, nr_failed = 0;
 	LIST_HEAD(folios);
 	LIST_HEAD(split_folios);
 	struct migrate_pages_stats astats;
+	struct migrate_control async_ctl;
 
-	if (mode == MIGRATE_ASYNC) {
+	if (ctl->mode == MIGRATE_ASYNC) {
 		rc = migrate_folios_batch(from, get_new_folio, put_new_folio,
-				private, mode, reason, ret_folios,
+				private, ctl, ret_folios,
 				&split_folios, stats,
 				NR_MAX_MIGRATE_PAGES_RETRY);
 		goto out;
 	}
 
+	async_ctl = *ctl;
+	async_ctl.mode = MIGRATE_ASYNC;
 	memset(&astats, 0, sizeof(astats));
 	/* Try to migrate in batch with MIGRATE_ASYNC mode firstly */
 	rc = migrate_folios_batch(from, get_new_folio, put_new_folio, private,
-			MIGRATE_ASYNC, reason, &folios, &split_folios, &astats,
+			&async_ctl, &folios, &split_folios, &astats,
 			NR_MAX_MIGRATE_ASYNC_RETRY);
 	stats->nr_succeeded += astats.nr_succeeded;
 	stats->nr_thp_succeeded += astats.nr_thp_succeeded;
@@ -2236,8 +2237,7 @@ static int __migrate_lru_folios(struct list_head *from, new_folio_t get_new_foli
 	while (!list_empty(from)) {
 		list_move(from->next, &folios);
 		rc = migrate_folios_batch(&folios, get_new_folio, put_new_folio,
-				private, mode, reason, ret_folios,
-				&split_folios, stats,
+				private, ctl, ret_folios, &split_folios, stats,
 				NR_MAX_MIGRATE_SYNC_RETRY);
 		list_splice_tail_init(&folios, ret_folios);
 		if (rc < 0)
@@ -2255,7 +2255,7 @@ static int __migrate_lru_folios(struct list_head *from, new_folio_t get_new_foli
 		 * folio they came from was already counted as one failure.
 		 */
 		migrate_folios_batch(&split_folios, get_new_folio, put_new_folio,
-				private, MIGRATE_ASYNC, reason, ret_folios,
+				private, &async_ctl, ret_folios,
 				NULL, stats, 1);
 		list_splice_tail_init(&split_folios, ret_folios);
 	}
@@ -2270,7 +2270,7 @@ static int __migrate_lru_folios(struct list_head *from, new_folio_t get_new_foli
  */
 static int migrate_lru_folios(struct list_head *from, new_folio_t get_new_folio,
 		free_folio_t put_new_folio, unsigned long private,
-		enum migrate_mode mode, enum migrate_reason reason,
+		const struct migrate_control *ctl,
 		struct list_head *ret_folios,
 		struct migrate_pages_stats *stats)
 {
@@ -2292,7 +2292,7 @@ static int migrate_lru_folios(struct list_head *from, new_folio_t get_new_folio,
 			list_splice_init(from, &folios);
 
 		rc = __migrate_lru_folios(&folios, get_new_folio, put_new_folio,
-				private, mode, reason, ret_folios, stats);
+				private, ctl, ret_folios, stats);
 		list_splice_tail_init(&folios, ret_folios);
 		if (rc < 0)
 			return rc;
@@ -2335,18 +2335,22 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 	int rc, rc_gather;
 	LIST_HEAD(ret_folios);
 	struct migrate_pages_stats stats;
+	const struct migrate_control ctl = {
+		.mode = mode,
+		.reason = reason,
+	};
 
-	trace_mm_migrate_pages_start(mode, reason);
+	trace_mm_migrate_pages_start(ctl.mode, ctl.reason);
 
 	memset(&stats, 0, sizeof(stats));
 
 	rc_gather = migrate_hugetlbs(from, get_new_folio, put_new_folio, private,
-				     mode, reason, &stats, &ret_folios);
+				     &ctl, &stats, &ret_folios);
 	if (rc_gather < 0)
 		goto out;
 
 	rc = migrate_movable_ops_pages(from, get_new_folio, put_new_folio,
-				       private, mode, reason, &stats,
+				       private, &ctl, &stats,
 				       &ret_folios);
 	if (rc < 0) {
 		rc_gather = rc;
@@ -2355,7 +2359,7 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 	rc_gather += rc;
 
 	rc = migrate_lru_folios(from, get_new_folio, put_new_folio,
-			private, mode, reason, &ret_folios, &stats);
+			private, &ctl, &ret_folios, &stats);
 	if (rc < 0) {
 		rc_gather = rc;
 		goto out;
@@ -2382,8 +2386,8 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 	count_vm_events(THP_MIGRATION_SPLIT, stats.nr_thp_split);
 	trace_mm_migrate_pages(stats.nr_succeeded, stats.nr_failed_pages,
 			       stats.nr_thp_succeeded, stats.nr_thp_failed,
-			       stats.nr_thp_split, stats.nr_split, mode,
-			       reason);
+			       stats.nr_thp_split, stats.nr_split, ctl.mode,
+			       ctl.reason);
 
 	if (ret_succeeded)
 		*ret_succeeded = stats.nr_succeeded;

-- 
2.43.0

Re: [sos-linux-ext-patches] [PATCH RFC 09/11] mm/migrate: thread migration policy through a control struct

[Garg, Shivank]

On Wed, 2026-09-02 at 10:52 +0000, Shivank Garg wrote:
> Migration mode and reason describe one migration invocation, but are passed
> 

[...]

> @@ -2189,27 +2188,29 @@ static int migrate_folios_batch(struct list_head *from,
>   */
>  static int __migrate_lru_folios(struct list_head *from, new_folio_t get_new_folio,
>  		free_folio_t put_new_folio, unsigned long private,
> -		enum migrate_mode mode, enum migrate_reason reason,
> -		struct list_head *ret_folios,
> +		const struct migrate_control *ctl, struct list_head *ret_folios,
>  		struct migrate_pages_stats *stats)
>  {
>  	int rc, nr_failed = 0;
>  	LIST_HEAD(folios);
>  	LIST_HEAD(split_folios);
>  	struct migrate_pages_stats astats;
> +	struct migrate_control async_ctl;
>  
> -	if (mode == MIGRATE_ASYNC) {
> +	if (ctl->mode == MIGRATE_ASYNC) {
>  		rc = migrate_folios_batch(from, get_new_folio, put_new_folio,
> -				private, mode, reason, ret_folios,
> +				private, ctl, ret_folios,
>  				&split_folios, stats,
>  				NR_MAX_MIGRATE_PAGES_RETRY);
>  		goto out;
>  	}
>  
> +	async_ctl = *ctl;
> +	async_ctl.mode = MIGRATE_ASYNC;
>  	memset(&astats, 0, sizeof(astats));
>  	/* Try to migrate in batch with MIGRATE_ASYNC mode firstly */



Sashiko:
Can this lead to an uninitialized stack variable being used?
If __migrate_lru_folios() is called with ctl->mode == MIGRATE_ASYNC, the
early check jumps directly to the out label, bypassing the initialization
of async_ctl. If large folios were split during that first pass, the
!list_empty(&split_folios) check will be true.
Could this cause migrate_folios_batch() to execute with garbage policy? For
example, it might interpret the uninitialized async_ctl.mode as MIGRATE_SYNC,
causing unintended blocking and sleeping during what should be an asynchronous
migration.
--

Yes, this is valid issue. I should initialize async_ctl early before it make
goto jump.

---
 mm/migrate.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/mm/migrate.c b/mm/migrate.c
index 731836664a86..c75300e00388 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -2199,8 +2199,9 @@ static int __migrate_lru_folios(struct list_head *from, new_folio_t get_new_foli
 	LIST_HEAD(folios);
 	LIST_HEAD(split_folios);
 	struct migrate_pages_stats astats;
-	struct migrate_control async_ctl;
+	struct migrate_control async_ctl = *ctl;
 
+	async_ctl.mode = MIGRATE_ASYNC;
 	if (ctl->mode == MIGRATE_ASYNC) {
 		rc = migrate_folios_batch(from, get_new_folio, put_new_folio,
 				private, ctl, ret_folios,
@@ -2209,8 +2210,6 @@ static int __migrate_lru_folios(struct list_head *from, new_folio_t get_new_foli
 		goto out;
 	}
 
-	async_ctl = *ctl;
-	async_ctl.mode = MIGRATE_ASYNC;
 	memset(&astats, 0, sizeof(astats));
 	/* Try to migrate in batch with MIGRATE_ASYNC mode firstly */
 	rc = migrate_folios_batch(from, get_new_folio, put_new_folio, private,
-- 
2.43.0

[PATCH RFC 10/11] mm/migrate: pass migrate_control to migrate_pages()

[Shivank Garg]

migrate_pages() still takes mode and reason separately and reconstructs a
migrate_control internally. This prevents callers from supplying additional
migration policy without extending its argument list.

Change migrate_pages() to take a const struct migrate_control pointer and
update callers to construct it on the stack. Derive asynchronous controls
by copying the caller's control and overriding only the mode, preserving
any additional policy fields.

No functional change intended.

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 include/linux/migrate.h |  5 ++---
 mm/compaction.c         |  8 ++++++--
 mm/damon/ops-common.c   |  7 +++++--
 mm/gup.c                |  7 +++++--
 mm/memory-failure.c     |  6 +++++-
 mm/memory_hotplug.c     |  6 +++++-
 mm/mempolicy.c          | 13 ++++++++++---
 mm/migrate.c            | 36 +++++++++++++++++++-----------------
 mm/page_alloc.c         |  6 +++++-
 mm/vmscan.c             |  7 +++++--
 10 files changed, 67 insertions(+), 34 deletions(-)

diff --git a/include/linux/migrate.h b/include/linux/migrate.h
index 78424b3824c2..cec5713944e3 100644
--- a/include/linux/migrate.h
+++ b/include/linux/migrate.h
@@ -57,8 +57,7 @@ void putback_movable_pages(struct list_head *l);
 int migrate_folio(struct address_space *mapping, struct folio *dst,
 		struct folio *src, enum migrate_mode mode);
 int migrate_pages(struct list_head *l, new_folio_t new, free_folio_t free,
-		unsigned long private, enum migrate_mode mode,
-		enum migrate_reason reason,
+		unsigned long private, const struct migrate_control *ctl,
 		unsigned int *ret_succeeded);
 struct folio *alloc_migration_target(struct folio *src, unsigned long private);
 bool isolate_movable_ops_page(struct page *page, isolate_mode_t mode);
@@ -78,7 +77,7 @@ int set_movable_ops(const struct movable_operations *ops, enum pagetype type);
 static inline void putback_movable_pages(struct list_head *l) {}
 static inline int migrate_pages(struct list_head *l, new_folio_t new,
 		free_folio_t free, unsigned long private,
-		enum migrate_mode mode, enum migrate_reason reason,
+		const struct migrate_control *ctl,
 		unsigned int *ret_succeeded)
 	{ return -ENOSYS; }
 static inline struct folio *alloc_migration_target(struct folio *src,
diff --git a/mm/compaction.c b/mm/compaction.c
index a049415512c6..e75fcb0ee007 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -2566,6 +2566,10 @@ compact_zone(struct compact_control *cc, struct capture_control *capc)
 	unsigned long end_pfn = zone_end_pfn(cc->zone);
 	unsigned long last_migrated_pfn;
 	const bool sync = cc->mode != MIGRATE_ASYNC;
+	const struct migrate_control ctl = {
+		.mode = cc->mode,
+		.reason = MR_COMPACTION,
+	};
 	bool update_cached;
 	unsigned int nr_succeeded = 0, nr_migratepages;
 	int order;
@@ -2696,8 +2700,8 @@ compact_zone(struct compact_control *cc, struct capture_control *capc)
 		 */
 		nr_migratepages = cc->nr_migratepages;
 		err = migrate_pages(&cc->migratepages, compaction_alloc,
-				compaction_free, (unsigned long)cc, cc->mode,
-				MR_COMPACTION, &nr_succeeded);
+				compaction_free, (unsigned long)cc, &ctl,
+				&nr_succeeded);
 
 		trace_mm_compaction_migratepages(nr_migratepages, nr_succeeded);
 
diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c
index fbda70d8ea4d..da848561aebb 100644
--- a/mm/damon/ops-common.c
+++ b/mm/damon/ops-common.c
@@ -315,6 +315,10 @@ static unsigned int __damon_migrate_folio_list(
 			__GFP_NOMEMALLOC | GFP_NOWAIT | __GFP_THISNODE,
 		.nid = target_nid,
 	};
+	const struct migrate_control ctl = {
+		.mode = MIGRATE_ASYNC,
+		.reason = MR_DAMON,
+	};
 
 	if (pgdat->node_id == target_nid || target_nid == NUMA_NO_NODE)
 		return 0;
@@ -324,8 +328,7 @@ static unsigned int __damon_migrate_folio_list(
 
 	/* Migration ignores all cpuset and mempolicy settings */
 	migrate_pages(migrate_folios, alloc_migration_target, NULL,
-		      (unsigned long)&mtc, MIGRATE_ASYNC, MR_DAMON,
-		      &nr_succeeded);
+		      (unsigned long)&mtc, &ctl, &nr_succeeded);
 
 	return nr_succeeded;
 }
diff --git a/mm/gup.c b/mm/gup.c
index eb898ea1ee22..73e1e8bfa4c0 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -2357,10 +2357,13 @@ migrate_longterm_unpinnable_folios(struct list_head *movable_folio_list,
 			.gfp_mask = GFP_USER | __GFP_NOWARN,
 			.reason = MR_LONGTERM_PIN,
 		};
+		const struct migrate_control ctl = {
+			.mode = MIGRATE_SYNC,
+			.reason = MR_LONGTERM_PIN,
+		};
 
 		if (migrate_pages(movable_folio_list, alloc_migration_target,
-				  NULL, (unsigned long)&mtc, MIGRATE_SYNC,
-				  MR_LONGTERM_PIN, NULL)) {
+				  NULL, (unsigned long)&mtc, &ctl, NULL)) {
 			ret = -ENOMEM;
 			goto err;
 		}
diff --git a/mm/memory-failure.c b/mm/memory-failure.c
index a8b03e2920ba..85d0a69a0844 100644
--- a/mm/memory-failure.c
+++ b/mm/memory-failure.c
@@ -2860,6 +2860,10 @@ static int soft_offline_in_use_page(struct page *page)
 		.gfp_mask = GFP_USER | __GFP_MOVABLE | __GFP_RETRY_MAYFAIL,
 		.reason = MR_MEMORY_FAILURE,
 	};
+	const struct migrate_control ctl = {
+		.mode = MIGRATE_SYNC,
+		.reason = MR_MEMORY_FAILURE,
+	};
 
 	if (!huge && folio_test_large(folio)) {
 		const int new_order = min_order_for_split(folio);
@@ -2916,7 +2920,7 @@ static int soft_offline_in_use_page(struct page *page)
 
 	if (isolated) {
 		ret = migrate_pages(&pagelist, alloc_migration_target, NULL,
-			(unsigned long)&mtc, MIGRATE_SYNC, MR_MEMORY_FAILURE, NULL);
+			(unsigned long)&mtc, &ctl, NULL);
 		if (!ret) {
 			bool release = !huge;
 
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 226ab9cb078a..1250b0d47702 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1901,6 +1901,10 @@ static void do_migrate_range(unsigned long start_pfn, unsigned long end_pfn)
 			.gfp_mask = GFP_KERNEL | __GFP_MOVABLE | __GFP_RETRY_MAYFAIL,
 			.reason = MR_MEMORY_HOTPLUG,
 		};
+		const struct migrate_control ctl = {
+			.mode = MIGRATE_SYNC,
+			.reason = MR_MEMORY_HOTPLUG,
+		};
 		int ret;
 
 		/*
@@ -1918,7 +1922,7 @@ static void do_migrate_range(unsigned long start_pfn, unsigned long end_pfn)
 		if (nodes_empty(nmask))
 			node_set(mtc.nid, nmask);
 		ret = migrate_pages(&source, alloc_migration_target, NULL,
-			(unsigned long)&mtc, MIGRATE_SYNC, MR_MEMORY_HOTPLUG, NULL);
+			(unsigned long)&mtc, &ctl, NULL);
 		if (ret) {
 			list_for_each_entry(folio, &source, lru) {
 				if (__ratelimit(&migrate_rs)) {
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 3498a5651d50..14f3f24a6fad 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -1295,6 +1295,10 @@ static long migrate_to_node(struct mm_struct *mm, int source, int dest,
 		.gfp_mask = GFP_HIGHUSER_MOVABLE | __GFP_THISNODE,
 		.reason = MR_SYSCALL,
 	};
+	const struct migrate_control ctl = {
+		.mode = MIGRATE_SYNC,
+		.reason = MR_SYSCALL,
+	};
 
 	nodes_clear(nmask);
 	node_set(source, nmask);
@@ -1320,7 +1324,7 @@ static long migrate_to_node(struct mm_struct *mm, int source, int dest,
 
 	if (!list_empty(&pagelist)) {
 		err = migrate_pages(&pagelist, alloc_migration_target, NULL,
-			(unsigned long)&mtc, MIGRATE_SYNC, MR_SYSCALL, NULL);
+			(unsigned long)&mtc, &ctl, NULL);
 		if (err)
 			putback_movable_pages(&pagelist);
 	}
@@ -1499,6 +1503,10 @@ static long do_mbind(unsigned long start, unsigned long len,
 	long err;
 	long nr_failed;
 	LIST_HEAD(pagelist);
+	const struct migrate_control ctl = {
+		.mode = MIGRATE_SYNC,
+		.reason = MR_MEMPOLICY_MBIND,
+	};
 
 	if (flags & ~(unsigned long)MPOL_MF_VALID)
 		return -EINVAL;
@@ -1616,8 +1624,7 @@ static long do_mbind(unsigned long start, unsigned long len,
 	if (!err && !list_empty(&pagelist)) {
 		nr_failed |= migrate_pages(&pagelist,
 				alloc_migration_target_by_mpol, NULL,
-				(unsigned long)&mmpol, MIGRATE_SYNC,
-				MR_MEMPOLICY_MBIND, NULL);
+				(unsigned long)&mmpol, &ctl, NULL);
 	}
 
 	if (nr_failed && (flags & MPOL_MF_STRICT))
diff --git a/mm/migrate.c b/mm/migrate.c
index e4bc7122076e..549f8b57092c 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -2312,9 +2312,8 @@ static int migrate_lru_folios(struct list_head *from, new_folio_t get_new_folio,
  * @put_new_folio:	The function used to free target folios if migration
  *			fails, or NULL if no special handling is necessary.
  * @private:		Private data to be passed on to get_new_folio()
- * @mode:		The migration mode that specifies the constraints for
- *			folio migration, if any.
- * @reason:		The reason for folio migration.
+ * @ctl:		The policy for this migration: blocking discipline,
+ *			and reason.
  * @ret_succeeded:	Set to the number of folios migrated successfully if
  *			the caller passes a non-NULL pointer.
  *
@@ -2330,27 +2329,23 @@ static int migrate_lru_folios(struct list_head *from, new_folio_t get_new_folio,
  */
 int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 		free_folio_t put_new_folio, unsigned long private,
-		enum migrate_mode mode, enum migrate_reason reason, unsigned int *ret_succeeded)
+		const struct migrate_control *ctl, unsigned int *ret_succeeded)
 {
 	int rc, rc_gather;
 	LIST_HEAD(ret_folios);
 	struct migrate_pages_stats stats;
-	const struct migrate_control ctl = {
-		.mode = mode,
-		.reason = reason,
-	};
 
-	trace_mm_migrate_pages_start(ctl.mode, ctl.reason);
+	trace_mm_migrate_pages_start(ctl->mode, ctl->reason);
 
 	memset(&stats, 0, sizeof(stats));
 
 	rc_gather = migrate_hugetlbs(from, get_new_folio, put_new_folio, private,
-				     &ctl, &stats, &ret_folios);
+				     ctl, &stats, &ret_folios);
 	if (rc_gather < 0)
 		goto out;
 
 	rc = migrate_movable_ops_pages(from, get_new_folio, put_new_folio,
-				       private, &ctl, &stats,
+				       private, ctl, &stats,
 				       &ret_folios);
 	if (rc < 0) {
 		rc_gather = rc;
@@ -2359,7 +2354,7 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 	rc_gather += rc;
 
 	rc = migrate_lru_folios(from, get_new_folio, put_new_folio,
-			private, &ctl, &ret_folios, &stats);
+			private, ctl, &ret_folios, &stats);
 	if (rc < 0) {
 		rc_gather = rc;
 		goto out;
@@ -2386,8 +2381,8 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 	count_vm_events(THP_MIGRATION_SPLIT, stats.nr_thp_split);
 	trace_mm_migrate_pages(stats.nr_succeeded, stats.nr_failed_pages,
 			       stats.nr_thp_succeeded, stats.nr_thp_failed,
-			       stats.nr_thp_split, stats.nr_split, ctl.mode,
-			       ctl.reason);
+			       stats.nr_thp_split, stats.nr_split, ctl->mode,
+			       ctl->reason);
 
 	if (ret_succeeded)
 		*ret_succeeded = stats.nr_succeeded;
@@ -2454,9 +2449,13 @@ static int do_move_pages_to_node(struct list_head *pagelist, int node)
 		.gfp_mask = GFP_HIGHUSER_MOVABLE | __GFP_THISNODE,
 		.reason = MR_SYSCALL,
 	};
+	const struct migrate_control ctl = {
+		.mode = MIGRATE_SYNC,
+		.reason = MR_SYSCALL,
+	};
 
 	err = migrate_pages(pagelist, alloc_migration_target, NULL,
-		(unsigned long)&mtc, MIGRATE_SYNC, MR_SYSCALL, NULL);
+		(unsigned long)&mtc, &ctl, NULL);
 	if (err)
 		putback_movable_pages(pagelist);
 	return err;
@@ -2961,11 +2960,14 @@ int migrate_misplaced_folio(struct folio *folio, int node)
 	LIST_HEAD(migratepages);
 	struct mem_cgroup *memcg = get_mem_cgroup_from_folio(folio);
 	struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat);
+	const struct migrate_control ctl = {
+		.mode = MIGRATE_ASYNC,
+		.reason = MR_NUMA_MISPLACED,
+	};
 
 	list_add(&folio->lru, &migratepages);
 	nr_remaining = migrate_pages(&migratepages, alloc_misplaced_dst_folio,
-				     NULL, node, MIGRATE_ASYNC,
-				     MR_NUMA_MISPLACED, &nr_succeeded);
+				     NULL, node, &ctl, &nr_succeeded);
 	if (nr_remaining && !list_empty(&migratepages))
 		putback_movable_pages(&migratepages);
 	if (nr_succeeded) {
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 12fac9084c48..8a921f6cdc86 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -7146,6 +7146,10 @@ static int __alloc_contig_migrate_range(struct compact_control *cc,
 		.gfp_mask = cc->gfp_mask,
 		.reason = MR_CONTIG_RANGE,
 	};
+	const struct migrate_control ctl = {
+		.mode = cc->mode,
+		.reason = MR_CONTIG_RANGE,
+	};
 
 	lru_cache_disable();
 
@@ -7172,7 +7176,7 @@ static int __alloc_contig_migrate_range(struct compact_control *cc,
 		cc->nr_migratepages -= nr_reclaimed;
 
 		ret = migrate_pages(&cc->migratepages, alloc_migration_target,
-			NULL, (unsigned long)&mtc, cc->mode, MR_CONTIG_RANGE, NULL);
+			NULL, (unsigned long)&mtc, &ctl, NULL);
 
 		/*
 		 * On -ENOMEM, migrate_pages() bails out right away. It is pointless
diff --git a/mm/vmscan.c b/mm/vmscan.c
index f11491ee9ed5..7e8b04c97342 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1013,6 +1013,10 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,
 		.nmask = &allowed_mask,
 		.reason = MR_DEMOTION,
 	};
+	const struct migrate_control ctl = {
+		.mode = MIGRATE_ASYNC,
+		.reason = MR_DEMOTION,
+	};
 
 	if (list_empty(demote_folios))
 		return 0;
@@ -1031,8 +1035,7 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,
 
 	/* Demotion ignores all cpuset and mempolicy settings */
 	migrate_pages(demote_folios, alloc_demote_folio, NULL,
-		      (unsigned long)&mtc, MIGRATE_ASYNC, MR_DEMOTION,
-		      &nr_succeeded);
+		      (unsigned long)&mtc, &ctl, &nr_succeeded);
 
 	return nr_succeeded;
 }

-- 
2.43.0

[PATCH RFC 11/11] mm/migrate: pass migrate_control to migrate_folio()

[Shivank Garg]

The migration core now carries policy in struct migrate_control, but
migrate_folio() and the address_space migrate_folio callback still receive
only the migration mode. This prevents additional policy, such as
copy-cache hints, from reaching folio-specific migration code.

Pass the control as const through migrate_folio(), the common migration
helpers, and the address_space callback. Update all implementations and
documentation accordingly. Existing callbacks continue to make their
decisions from ctl->mode only, none changes behavior yet.

No functional change intended.

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 Documentation/filesystems/locking.rst |  2 +-
 Documentation/filesystems/vfs.rst     |  7 +++--
 fs/aio.c                              |  2 +-
 fs/btrfs/disk-io.c                    |  5 ++--
 fs/btrfs/inode.c                      |  6 ++---
 fs/hugetlbfs/inode.c                  |  4 +--
 fs/jfs/jfs_metapage.c                 | 14 +++++-----
 fs/nfs/internal.h                     |  2 +-
 fs/nfs/write.c                        |  8 +++---
 include/linux/buffer_head.h           |  6 +++--
 include/linux/fs.h                    |  6 ++---
 include/linux/migrate.h               |  2 +-
 include/linux/pagemap.h               |  2 +-
 mm/migrate.c                          | 50 +++++++++++++++++++----------------
 mm/secretmem.c                        |  3 ++-
 virt/kvm/guest_memfd.c                |  2 +-
 16 files changed, 66 insertions(+), 55 deletions(-)

diff --git a/Documentation/filesystems/locking.rst b/Documentation/filesystems/locking.rst
index 844d65eb47a5..b406df403b4a 100644
--- a/Documentation/filesystems/locking.rst
+++ b/Documentation/filesystems/locking.rst
@@ -260,7 +260,7 @@ prototypes::
 	void (*free_folio)(struct folio *);
 	int (*direct_IO)(struct kiocb *, struct iov_iter *iter);
 	int (*migrate_folio)(struct address_space *, struct folio *dst,
-			struct folio *src, enum migrate_mode);
+			struct folio *src, const struct migrate_control *ctl);
 	int (*launder_folio)(struct folio *);
 	bool (*is_partially_uptodate)(struct folio *, size_t from, size_t count);
 	int (*error_remove_folio)(struct address_space *, struct folio *);
diff --git a/Documentation/filesystems/vfs.rst b/Documentation/filesystems/vfs.rst
index d3a93eec3945..8e1404723db1 100644
--- a/Documentation/filesystems/vfs.rst
+++ b/Documentation/filesystems/vfs.rst
@@ -767,7 +767,8 @@ cache in your filesystem.  The following members are defined:
 		void (*free_folio)(struct folio *);
 		ssize_t (*direct_IO)(struct kiocb *, struct iov_iter *iter);
 		int (*migrate_folio)(struct mapping *, struct folio *dst,
-				struct folio *src, enum migrate_mode);
+				struct folio *src,
+				const struct migrate_control *ctl);
 		int (*launder_folio) (struct folio *);
 
 		bool (*is_partially_uptodate) (struct folio *, size_t from,
@@ -939,7 +940,9 @@ cache in your filesystem.  The following members are defined:
 	wants to relocate a folio (maybe from a memory device that is
 	signalling imminent failure) it will pass a new folio and an old
 	folio to this function.  migrate_folio should transfer any private
-	data across and update any references that it has to the folio.
+	data across and update any references that it has to the folio.  The
+	control describes the blocking mode and the reason for migration.  If
+	the mode is ``MIGRATE_ASYNC``, the callback must not block.
 
 ``launder_folio``
 	Called before freeing a folio - it writes back the dirty folio.
diff --git a/fs/aio.c b/fs/aio.c
index d78acc69f487..58ee9d0845dd 100644
--- a/fs/aio.c
+++ b/fs/aio.c
@@ -458,7 +458,7 @@ static const struct file_operations aio_ring_fops = {
 
 #if IS_ENABLED(CONFIG_MIGRATION)
 static int aio_migrate_folio(struct address_space *mapping, struct folio *dst,
-			struct folio *src, enum migrate_mode mode)
+			     struct folio *src, const struct migrate_control *ctl)
 {
 	struct kioctx *ctx;
 	struct aio_inode_info *ai = AIO_I(mapping->host);
diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
index 819727460bcf..509651577a1c 100644
--- a/fs/btrfs/disk-io.c
+++ b/fs/btrfs/disk-io.c
@@ -460,7 +460,8 @@ int btrfs_validate_extent_buffer(struct extent_buffer *eb,
 
 #ifdef CONFIG_MIGRATION
 static int btree_migrate_folio(struct address_space *mapping,
-		struct folio *dst, struct folio *src, enum migrate_mode mode)
+		struct folio *dst, struct folio *src,
+		const struct migrate_control *ctl)
 {
 	/*
 	 * we can't safely write a btree page from here,
@@ -475,7 +476,7 @@ static int btree_migrate_folio(struct address_space *mapping,
 	if (folio_get_private(src) &&
 	    !filemap_release_folio(src, GFP_KERNEL))
 		return -EAGAIN;
-	return migrate_folio(mapping, dst, src, mode);
+	return migrate_folio(mapping, dst, src, ctl);
 }
 #else
 #define btree_migrate_folio NULL
diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index 3c10a0ef0002..9f1e567fa5ed 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -7653,10 +7653,10 @@ static bool btrfs_release_folio(struct folio *folio, gfp_t gfp_flags)
 
 #ifdef CONFIG_MIGRATION
 static int btrfs_migrate_folio(struct address_space *mapping,
-			     struct folio *dst, struct folio *src,
-			     enum migrate_mode mode)
+			       struct folio *dst, struct folio *src,
+			       const struct migrate_control *ctl)
 {
-	int ret = filemap_migrate_folio(mapping, dst, src, mode);
+	int ret = filemap_migrate_folio(mapping, dst, src, ctl);
 
 	if (ret)
 		return ret;
diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index 7611a8470ea2..367148251689 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -1022,8 +1022,8 @@ static int hugetlbfs_symlink(struct mnt_idmap *idmap,
 
 #ifdef CONFIG_MIGRATION
 static int hugetlbfs_migrate_folio(struct address_space *mapping,
-				struct folio *dst, struct folio *src,
-				enum migrate_mode mode)
+				   struct folio *dst, struct folio *src,
+				   const struct migrate_control *ctl)
 {
 	int rc;
 
diff --git a/fs/jfs/jfs_metapage.c b/fs/jfs/jfs_metapage.c
index 41fe12e641ce..7958edacb497 100644
--- a/fs/jfs/jfs_metapage.c
+++ b/fs/jfs/jfs_metapage.c
@@ -155,7 +155,7 @@ static inline void dec_io(struct folio *folio, blk_status_t status,
 #ifdef CONFIG_MIGRATION
 static int __metapage_migrate_folio(struct address_space *mapping,
 				    struct folio *dst, struct folio *src,
-				    enum migrate_mode mode)
+				    const struct migrate_control *ctl)
 {
 	struct meta_anchor *src_anchor = src->private;
 	struct metapage *mps[MPS_PER_PAGE] = {0};
@@ -168,7 +168,7 @@ static int __metapage_migrate_folio(struct address_space *mapping,
 			return -EAGAIN;
 	}
 
-	rc = filemap_migrate_folio(mapping, dst, src, mode);
+	rc = filemap_migrate_folio(mapping, dst, src, ctl);
 	if (rc)
 		return rc;
 
@@ -231,7 +231,7 @@ static inline void remove_metapage(struct folio *folio, struct metapage *mp)
 #ifdef CONFIG_MIGRATION
 static int __metapage_migrate_folio(struct address_space *mapping,
 				    struct folio *dst, struct folio *src,
-				    enum migrate_mode mode)
+				    const struct migrate_control *ctl)
 {
 	struct metapage *mp;
 	int page_offset;
@@ -241,7 +241,7 @@ static int __metapage_migrate_folio(struct address_space *mapping,
 	if (metapage_locked(mp))
 		return -EAGAIN;
 
-	rc = filemap_migrate_folio(mapping, dst, src, mode);
+	rc = filemap_migrate_folio(mapping, dst, src, ctl);
 	if (rc)
 		return rc;
 
@@ -645,18 +645,18 @@ static bool metapage_release_folio(struct folio *folio, gfp_t gfp_mask)
  */
 static int metapage_migrate_folio(struct address_space *mapping,
 				  struct folio *dst, struct folio *src,
-				  enum migrate_mode mode)
+				  const struct migrate_control *ctl)
 {
 	int expected_count;
 
 	if (!src->private)
-		return filemap_migrate_folio(mapping, dst, src, mode);
+		return filemap_migrate_folio(mapping, dst, src, ctl);
 
 	/* Check whether page does not have extra refs before we do more work */
 	expected_count = folio_expected_ref_count(src) + 1;
 	if (folio_ref_count(src) != expected_count)
 		return -EAGAIN;
-	return __metapage_migrate_folio(mapping, dst, src, mode);
+	return __metapage_migrate_folio(mapping, dst, src, ctl);
 }
 #else
 #define metapage_migrate_folio NULL
diff --git a/fs/nfs/internal.h b/fs/nfs/internal.h
index abc81f5ae578..e1f366cff2c0 100644
--- a/fs/nfs/internal.h
+++ b/fs/nfs/internal.h
@@ -673,7 +673,7 @@ void nfs_clear_pnfs_ds_commit_verifiers(struct pnfs_ds_commit_info *cinfo)
 
 #ifdef CONFIG_MIGRATION
 int nfs_migrate_folio(struct address_space *, struct folio *dst,
-		struct folio *src, enum migrate_mode);
+		struct folio *src, const struct migrate_control *);
 #else
 #define nfs_migrate_folio NULL
 #endif
diff --git a/fs/nfs/write.c b/fs/nfs/write.c
index 623e7ef1f73d..159135900399 100644
--- a/fs/nfs/write.c
+++ b/fs/nfs/write.c
@@ -2112,7 +2112,7 @@ int nfs_wb_folio(struct inode *inode, struct folio *folio)
 
 #ifdef CONFIG_MIGRATION
 int nfs_migrate_folio(struct address_space *mapping, struct folio *dst,
-		struct folio *src, enum migrate_mode mode)
+		struct folio *src, const struct migrate_control *ctl)
 {
 	/*
 	 * If the private flag is set, the folio is currently associated with
@@ -2123,19 +2123,19 @@ int nfs_migrate_folio(struct address_space *mapping, struct folio *dst,
 	 *        the folio lock.
 	 */
 	if (folio_test_private(src)) {
-		if (mode == MIGRATE_SYNC)
+		if (ctl->mode == MIGRATE_SYNC)
 			nfs_wb_folio(src->mapping->host, src);
 		if (folio_test_private(src))
 			return -EBUSY;
 	}
 
 	if (folio_test_private_2(src)) { /* [DEPRECATED] */
-		if (mode == MIGRATE_ASYNC)
+		if (ctl->mode == MIGRATE_ASYNC)
 			return -EBUSY;
 		folio_wait_private_2(src);
 	}
 
-	return migrate_folio(mapping, dst, src, mode);
+	return migrate_folio(mapping, dst, src, ctl);
 }
 #endif
 
diff --git a/include/linux/buffer_head.h b/include/linux/buffer_head.h
index fd2c7115c054..e4476596c9b6 100644
--- a/include/linux/buffer_head.h
+++ b/include/linux/buffer_head.h
@@ -277,9 +277,11 @@ int block_truncate_page(struct address_space *, loff_t, get_block_t *);
 
 #ifdef CONFIG_MIGRATION
 extern int buffer_migrate_folio(struct address_space *,
-		struct folio *dst, struct folio *src, enum migrate_mode);
+		struct folio *dst, struct folio *src,
+		const struct migrate_control *);
 extern int buffer_migrate_folio_norefs(struct address_space *,
-		struct folio *dst, struct folio *src, enum migrate_mode);
+		struct folio *dst, struct folio *src,
+		const struct migrate_control *);
 #else
 #define buffer_migrate_folio NULL
 #define buffer_migrate_folio_norefs NULL
diff --git a/include/linux/fs.h b/include/linux/fs.h
index f9d1e05e8ae6..93ba6054c478 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -423,11 +423,11 @@ struct address_space_operations {
 	void (*free_folio)(struct folio *folio);
 	ssize_t (*direct_IO)(struct kiocb *, struct iov_iter *iter);
 	/*
-	 * migrate the contents of a folio to the specified target. If
-	 * migrate_mode is MIGRATE_ASYNC, it must not block.
+	 * Migrate the contents of a folio to the specified target.  The
+	 * callback must not block when ctl->mode is MIGRATE_ASYNC.
 	 */
 	int (*migrate_folio)(struct address_space *, struct folio *dst,
-			struct folio *src, enum migrate_mode);
+			struct folio *src, const struct migrate_control *ctl);
 	int (*launder_folio)(struct folio *);
 	bool (*is_partially_uptodate) (struct folio *, size_t from,
 			size_t count);
diff --git a/include/linux/migrate.h b/include/linux/migrate.h
index cec5713944e3..123111e7abd3 100644
--- a/include/linux/migrate.h
+++ b/include/linux/migrate.h
@@ -55,7 +55,7 @@ extern const char *migrate_reason_names[MR_TYPES];
 
 void putback_movable_pages(struct list_head *l);
 int migrate_folio(struct address_space *mapping, struct folio *dst,
-		struct folio *src, enum migrate_mode mode);
+		struct folio *src, const struct migrate_control *ctl);
 int migrate_pages(struct list_head *l, new_folio_t new, free_folio_t free,
 		unsigned long private, const struct migrate_control *ctl,
 		unsigned int *ret_succeeded);
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 0adfa6605653..455d57307f88 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -1327,7 +1327,7 @@ bool noop_dirty_folio(struct address_space *mapping, struct folio *folio);
 
 #ifdef CONFIG_MIGRATION
 int filemap_migrate_folio(struct address_space *mapping, struct folio *dst,
-		struct folio *src, enum migrate_mode mode);
+		struct folio *src, const struct migrate_control *ctl);
 #else
 #define filemap_migrate_folio NULL
 #endif
diff --git a/mm/migrate.c b/mm/migrate.c
index 549f8b57092c..731836664a86 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -869,7 +869,7 @@ EXPORT_SYMBOL(folio_migrate_flags);
 
 static int __migrate_folio(struct address_space *mapping, struct folio *dst,
 			   struct folio *src, void *src_private,
-			   enum migrate_mode mode)
+			   const struct migrate_control *ctl)
 {
 	int rc, expected_count = folio_expected_ref_count(src) + 1;
 
@@ -897,7 +897,7 @@ static int __migrate_folio(struct address_space *mapping, struct folio *dst,
  * @mapping: The address_space containing the folio.
  * @dst: The folio to migrate the data to.
  * @src: The folio containing the current data.
- * @mode: How to migrate the folio.
+ * @ctl: Migration policy for this folio.
  *
  * Common logic to directly migrate a single LRU folio suitable for
  * folios that do not have private data.
@@ -905,10 +905,10 @@ static int __migrate_folio(struct address_space *mapping, struct folio *dst,
  * Folios are locked upon entry and exit.
  */
 int migrate_folio(struct address_space *mapping, struct folio *dst,
-		  struct folio *src, enum migrate_mode mode)
+		  struct folio *src, const struct migrate_control *ctl)
 {
 	BUG_ON(folio_test_writeback(src));	/* Writeback must be complete */
-	return __migrate_folio(mapping, dst, src, NULL, mode);
+	return __migrate_folio(mapping, dst, src, NULL, ctl);
 }
 EXPORT_SYMBOL(migrate_folio);
 
@@ -947,8 +947,8 @@ static bool buffer_migrate_lock_buffers(struct buffer_head *head,
 }
 
 static int __buffer_migrate_folio(struct address_space *mapping,
-		struct folio *dst, struct folio *src, enum migrate_mode mode,
-		bool check_refs)
+		struct folio *dst, struct folio *src,
+		const struct migrate_control *ctl, bool check_refs)
 {
 	struct buffer_head *bh, *head;
 	int rc;
@@ -956,14 +956,14 @@ static int __buffer_migrate_folio(struct address_space *mapping,
 
 	head = folio_buffers(src);
 	if (!head)
-		return migrate_folio(mapping, dst, src, mode);
+		return migrate_folio(mapping, dst, src, ctl);
 
 	/* Check whether page does not have extra refs before we do more work */
 	expected_count = folio_expected_ref_count(src) + 1;
 	if (folio_ref_count(src) != expected_count)
 		return -EAGAIN;
 
-	if (!buffer_migrate_lock_buffers(head, mode))
+	if (!buffer_migrate_lock_buffers(head, ctl->mode))
 		return -EAGAIN;
 
 	if (check_refs) {
@@ -995,7 +995,7 @@ static int __buffer_migrate_folio(struct address_space *mapping,
 		}
 	}
 
-	rc = filemap_migrate_folio(mapping, dst, src, mode);
+	rc = filemap_migrate_folio(mapping, dst, src, ctl);
 	if (rc)
 		goto unlock_buffers;
 
@@ -1022,7 +1022,7 @@ static int __buffer_migrate_folio(struct address_space *mapping,
  * @mapping: The address space containing @src.
  * @dst: The folio to migrate to.
  * @src: The folio to migrate from.
- * @mode: How to migrate the folio.
+ * @ctl: Migration policy for this folio.
  *
  * This function can only be used if the underlying filesystem guarantees
  * that no other references to @src exist. For example attached buffer
@@ -1033,9 +1033,10 @@ static int __buffer_migrate_folio(struct address_space *mapping,
  * Return: 0 on success or a negative errno on failure.
  */
 int buffer_migrate_folio(struct address_space *mapping,
-		struct folio *dst, struct folio *src, enum migrate_mode mode)
+		struct folio *dst, struct folio *src,
+		const struct migrate_control *ctl)
 {
-	return __buffer_migrate_folio(mapping, dst, src, mode, false);
+	return __buffer_migrate_folio(mapping, dst, src, ctl, false);
 }
 EXPORT_SYMBOL(buffer_migrate_folio);
 
@@ -1044,7 +1045,7 @@ EXPORT_SYMBOL(buffer_migrate_folio);
  * @mapping: The address space containing @src.
  * @dst: The folio to migrate to.
  * @src: The folio to migrate from.
- * @mode: How to migrate the folio.
+ * @ctl: Migration policy for this folio.
  *
  * Like buffer_migrate_folio() except that this variant is more careful
  * and checks that there are also no buffer head references. This function
@@ -1054,17 +1055,19 @@ EXPORT_SYMBOL(buffer_migrate_folio);
  * Return: 0 on success or a negative errno on failure.
  */
 int buffer_migrate_folio_norefs(struct address_space *mapping,
-		struct folio *dst, struct folio *src, enum migrate_mode mode)
+		struct folio *dst, struct folio *src,
+		const struct migrate_control *ctl)
 {
-	return __buffer_migrate_folio(mapping, dst, src, mode, true);
+	return __buffer_migrate_folio(mapping, dst, src, ctl, true);
 }
 EXPORT_SYMBOL_GPL(buffer_migrate_folio_norefs);
 #endif /* CONFIG_BUFFER_HEAD */
 
 int filemap_migrate_folio(struct address_space *mapping,
-		struct folio *dst, struct folio *src, enum migrate_mode mode)
+		struct folio *dst, struct folio *src,
+		const struct migrate_control *ctl)
 {
-	return __migrate_folio(mapping, dst, src, folio_get_private(src), mode);
+	return __migrate_folio(mapping, dst, src, folio_get_private(src), ctl);
 }
 EXPORT_SYMBOL_GPL(filemap_migrate_folio);
 
@@ -1072,7 +1075,8 @@ EXPORT_SYMBOL_GPL(filemap_migrate_folio);
  * Default handling if a filesystem does not provide a migration function.
  */
 static int fallback_migrate_folio(struct address_space *mapping,
-		struct folio *dst, struct folio *src, enum migrate_mode mode)
+		struct folio *dst, struct folio *src,
+		const struct migrate_control *ctl)
 {
 	WARN_ONCE(mapping->a_ops->writepages,
 			"%ps does not implement migrate_folio\n",
@@ -1085,9 +1089,9 @@ static int fallback_migrate_folio(struct address_space *mapping,
 	 * can't migrate automatically.
 	 */
 	if (!filemap_release_folio(src, GFP_KERNEL))
-		return mode == MIGRATE_SYNC ? -EAGAIN : -EBUSY;
+		return ctl->mode == MIGRATE_SYNC ? -EAGAIN : -EBUSY;
 
-	return migrate_folio(mapping, dst, src, mode);
+	return migrate_folio(mapping, dst, src, ctl);
 }
 
 /*
@@ -1112,7 +1116,7 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,
 	VM_BUG_ON_FOLIO(!folio_test_locked(dst), dst);
 
 	if (!mapping)
-		rc = migrate_folio(mapping, dst, src, ctl->mode);
+		rc = migrate_folio(mapping, dst, src, ctl);
 	else if (mapping_inaccessible(mapping))
 		rc = -EOPNOTSUPP;
 	else if (mapping->a_ops->migrate_folio)
@@ -1123,9 +1127,9 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,
 		 * migrate_folio callback. This is the most common path
 		 * for page migration.
 		 */
-		rc = mapping->a_ops->migrate_folio(mapping, dst, src, ctl->mode);
+		rc = mapping->a_ops->migrate_folio(mapping, dst, src, ctl);
 	else
-		rc = fallback_migrate_folio(mapping, dst, src, ctl->mode);
+		rc = fallback_migrate_folio(mapping, dst, src, ctl);
 
 	if (!rc) {
 		/*
diff --git a/mm/secretmem.c b/mm/secretmem.c
index d29865075b6e..5c81833981d2 100644
--- a/mm/secretmem.c
+++ b/mm/secretmem.c
@@ -144,7 +144,8 @@ static const struct file_operations secretmem_fops = {
 };
 
 static int secretmem_migrate_folio(struct address_space *mapping,
-		struct folio *dst, struct folio *src, enum migrate_mode mode)
+		struct folio *dst, struct folio *src,
+		const struct migrate_control *ctl)
 {
 	return -EBUSY;
 }
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 625e62e1a031..0a9af8c4768d 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -457,7 +457,7 @@ static struct file_operations kvm_gmem_fops = {
 
 static int kvm_gmem_migrate_folio(struct address_space *mapping,
 				  struct folio *dst, struct folio *src,
-				  enum migrate_mode mode)
+				  const struct migrate_control *ctl)
 {
 	WARN_ON_ONCE(1);
 	return -EINVAL;

-- 
2.43.0

Re: [PATCH RFC 11/11] mm/migrate: pass migrate_control to migrate_folio()

[Jan Kara]

On Wed 02-09-26 10:52:25, Shivank Garg wrote:
> The migration core now carries policy in struct migrate_control, but
> migrate_folio() and the address_space migrate_folio callback still receive
> only the migration mode. This prevents additional policy, such as
> copy-cache hints, from reaching folio-specific migration code.
> 
> Pass the control as const through migrate_folio(), the common migration
> helpers, and the address_space callback. Update all implementations and
> documentation accordingly. Existing callbacks continue to make their
> decisions from ctl->mode only, none changes behavior yet.
> 
> No functional change intended.
> 
> Signed-off-by: Shivank Garg <shivankg@amd.com>

The changes to FS interface look good to me. Feel free to add:

Reviewed-by: Jan Kara <jack@suse.cz>

								Honza

> ---
>  Documentation/filesystems/locking.rst |  2 +-
>  Documentation/filesystems/vfs.rst     |  7 +++--
>  fs/aio.c                              |  2 +-
>  fs/btrfs/disk-io.c                    |  5 ++--
>  fs/btrfs/inode.c                      |  6 ++---
>  fs/hugetlbfs/inode.c                  |  4 +--
>  fs/jfs/jfs_metapage.c                 | 14 +++++-----
>  fs/nfs/internal.h                     |  2 +-
>  fs/nfs/write.c                        |  8 +++---
>  include/linux/buffer_head.h           |  6 +++--
>  include/linux/fs.h                    |  6 ++---
>  include/linux/migrate.h               |  2 +-
>  include/linux/pagemap.h               |  2 +-
>  mm/migrate.c                          | 50 +++++++++++++++++++----------------
>  mm/secretmem.c                        |  3 ++-
>  virt/kvm/guest_memfd.c                |  2 +-
>  16 files changed, 66 insertions(+), 55 deletions(-)
> 
> diff --git a/Documentation/filesystems/locking.rst b/Documentation/filesystems/locking.rst
> index 844d65eb47a5..b406df403b4a 100644
> --- a/Documentation/filesystems/locking.rst
> +++ b/Documentation/filesystems/locking.rst
> @@ -260,7 +260,7 @@ prototypes::
>  	void (*free_folio)(struct folio *);
>  	int (*direct_IO)(struct kiocb *, struct iov_iter *iter);
>  	int (*migrate_folio)(struct address_space *, struct folio *dst,
> -			struct folio *src, enum migrate_mode);
> +			struct folio *src, const struct migrate_control *ctl);
>  	int (*launder_folio)(struct folio *);
>  	bool (*is_partially_uptodate)(struct folio *, size_t from, size_t count);
>  	int (*error_remove_folio)(struct address_space *, struct folio *);
> diff --git a/Documentation/filesystems/vfs.rst b/Documentation/filesystems/vfs.rst
> index d3a93eec3945..8e1404723db1 100644
> --- a/Documentation/filesystems/vfs.rst
> +++ b/Documentation/filesystems/vfs.rst
> @@ -767,7 +767,8 @@ cache in your filesystem.  The following members are defined:
>  		void (*free_folio)(struct folio *);
>  		ssize_t (*direct_IO)(struct kiocb *, struct iov_iter *iter);
>  		int (*migrate_folio)(struct mapping *, struct folio *dst,
> -				struct folio *src, enum migrate_mode);
> +				struct folio *src,
> +				const struct migrate_control *ctl);
>  		int (*launder_folio) (struct folio *);
>  
>  		bool (*is_partially_uptodate) (struct folio *, size_t from,
> @@ -939,7 +940,9 @@ cache in your filesystem.  The following members are defined:
>  	wants to relocate a folio (maybe from a memory device that is
>  	signalling imminent failure) it will pass a new folio and an old
>  	folio to this function.  migrate_folio should transfer any private
> -	data across and update any references that it has to the folio.
> +	data across and update any references that it has to the folio.  The
> +	control describes the blocking mode and the reason for migration.  If
> +	the mode is ``MIGRATE_ASYNC``, the callback must not block.
>  
>  ``launder_folio``
>  	Called before freeing a folio - it writes back the dirty folio.
> diff --git a/fs/aio.c b/fs/aio.c
> index d78acc69f487..58ee9d0845dd 100644
> --- a/fs/aio.c
> +++ b/fs/aio.c
> @@ -458,7 +458,7 @@ static const struct file_operations aio_ring_fops = {
>  
>  #if IS_ENABLED(CONFIG_MIGRATION)
>  static int aio_migrate_folio(struct address_space *mapping, struct folio *dst,
> -			struct folio *src, enum migrate_mode mode)
> +			     struct folio *src, const struct migrate_control *ctl)
>  {
>  	struct kioctx *ctx;
>  	struct aio_inode_info *ai = AIO_I(mapping->host);
> diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
> index 819727460bcf..509651577a1c 100644
> --- a/fs/btrfs/disk-io.c
> +++ b/fs/btrfs/disk-io.c
> @@ -460,7 +460,8 @@ int btrfs_validate_extent_buffer(struct extent_buffer *eb,
>  
>  #ifdef CONFIG_MIGRATION
>  static int btree_migrate_folio(struct address_space *mapping,
> -		struct folio *dst, struct folio *src, enum migrate_mode mode)
> +		struct folio *dst, struct folio *src,
> +		const struct migrate_control *ctl)
>  {
>  	/*
>  	 * we can't safely write a btree page from here,
> @@ -475,7 +476,7 @@ static int btree_migrate_folio(struct address_space *mapping,
>  	if (folio_get_private(src) &&
>  	    !filemap_release_folio(src, GFP_KERNEL))
>  		return -EAGAIN;
> -	return migrate_folio(mapping, dst, src, mode);
> +	return migrate_folio(mapping, dst, src, ctl);
>  }
>  #else
>  #define btree_migrate_folio NULL
> diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
> index 3c10a0ef0002..9f1e567fa5ed 100644
> --- a/fs/btrfs/inode.c
> +++ b/fs/btrfs/inode.c
> @@ -7653,10 +7653,10 @@ static bool btrfs_release_folio(struct folio *folio, gfp_t gfp_flags)
>  
>  #ifdef CONFIG_MIGRATION
>  static int btrfs_migrate_folio(struct address_space *mapping,
> -			     struct folio *dst, struct folio *src,
> -			     enum migrate_mode mode)
> +			       struct folio *dst, struct folio *src,
> +			       const struct migrate_control *ctl)
>  {
> -	int ret = filemap_migrate_folio(mapping, dst, src, mode);
> +	int ret = filemap_migrate_folio(mapping, dst, src, ctl);
>  
>  	if (ret)
>  		return ret;
> diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
> index 7611a8470ea2..367148251689 100644
> --- a/fs/hugetlbfs/inode.c
> +++ b/fs/hugetlbfs/inode.c
> @@ -1022,8 +1022,8 @@ static int hugetlbfs_symlink(struct mnt_idmap *idmap,
>  
>  #ifdef CONFIG_MIGRATION
>  static int hugetlbfs_migrate_folio(struct address_space *mapping,
> -				struct folio *dst, struct folio *src,
> -				enum migrate_mode mode)
> +				   struct folio *dst, struct folio *src,
> +				   const struct migrate_control *ctl)
>  {
>  	int rc;
>  
> diff --git a/fs/jfs/jfs_metapage.c b/fs/jfs/jfs_metapage.c
> index 41fe12e641ce..7958edacb497 100644
> --- a/fs/jfs/jfs_metapage.c
> +++ b/fs/jfs/jfs_metapage.c
> @@ -155,7 +155,7 @@ static inline void dec_io(struct folio *folio, blk_status_t status,
>  #ifdef CONFIG_MIGRATION
>  static int __metapage_migrate_folio(struct address_space *mapping,
>  				    struct folio *dst, struct folio *src,
> -				    enum migrate_mode mode)
> +				    const struct migrate_control *ctl)
>  {
>  	struct meta_anchor *src_anchor = src->private;
>  	struct metapage *mps[MPS_PER_PAGE] = {0};
> @@ -168,7 +168,7 @@ static int __metapage_migrate_folio(struct address_space *mapping,
>  			return -EAGAIN;
>  	}
>  
> -	rc = filemap_migrate_folio(mapping, dst, src, mode);
> +	rc = filemap_migrate_folio(mapping, dst, src, ctl);
>  	if (rc)
>  		return rc;
>  
> @@ -231,7 +231,7 @@ static inline void remove_metapage(struct folio *folio, struct metapage *mp)
>  #ifdef CONFIG_MIGRATION
>  static int __metapage_migrate_folio(struct address_space *mapping,
>  				    struct folio *dst, struct folio *src,
> -				    enum migrate_mode mode)
> +				    const struct migrate_control *ctl)
>  {
>  	struct metapage *mp;
>  	int page_offset;
> @@ -241,7 +241,7 @@ static int __metapage_migrate_folio(struct address_space *mapping,
>  	if (metapage_locked(mp))
>  		return -EAGAIN;
>  
> -	rc = filemap_migrate_folio(mapping, dst, src, mode);
> +	rc = filemap_migrate_folio(mapping, dst, src, ctl);
>  	if (rc)
>  		return rc;
>  
> @@ -645,18 +645,18 @@ static bool metapage_release_folio(struct folio *folio, gfp_t gfp_mask)
>   */
>  static int metapage_migrate_folio(struct address_space *mapping,
>  				  struct folio *dst, struct folio *src,
> -				  enum migrate_mode mode)
> +				  const struct migrate_control *ctl)
>  {
>  	int expected_count;
>  
>  	if (!src->private)
> -		return filemap_migrate_folio(mapping, dst, src, mode);
> +		return filemap_migrate_folio(mapping, dst, src, ctl);
>  
>  	/* Check whether page does not have extra refs before we do more work */
>  	expected_count = folio_expected_ref_count(src) + 1;
>  	if (folio_ref_count(src) != expected_count)
>  		return -EAGAIN;
> -	return __metapage_migrate_folio(mapping, dst, src, mode);
> +	return __metapage_migrate_folio(mapping, dst, src, ctl);
>  }
>  #else
>  #define metapage_migrate_folio NULL
> diff --git a/fs/nfs/internal.h b/fs/nfs/internal.h
> index abc81f5ae578..e1f366cff2c0 100644
> --- a/fs/nfs/internal.h
> +++ b/fs/nfs/internal.h
> @@ -673,7 +673,7 @@ void nfs_clear_pnfs_ds_commit_verifiers(struct pnfs_ds_commit_info *cinfo)
>  
>  #ifdef CONFIG_MIGRATION
>  int nfs_migrate_folio(struct address_space *, struct folio *dst,
> -		struct folio *src, enum migrate_mode);
> +		struct folio *src, const struct migrate_control *);
>  #else
>  #define nfs_migrate_folio NULL
>  #endif
> diff --git a/fs/nfs/write.c b/fs/nfs/write.c
> index 623e7ef1f73d..159135900399 100644
> --- a/fs/nfs/write.c
> +++ b/fs/nfs/write.c
> @@ -2112,7 +2112,7 @@ int nfs_wb_folio(struct inode *inode, struct folio *folio)
>  
>  #ifdef CONFIG_MIGRATION
>  int nfs_migrate_folio(struct address_space *mapping, struct folio *dst,
> -		struct folio *src, enum migrate_mode mode)
> +		struct folio *src, const struct migrate_control *ctl)
>  {
>  	/*
>  	 * If the private flag is set, the folio is currently associated with
> @@ -2123,19 +2123,19 @@ int nfs_migrate_folio(struct address_space *mapping, struct folio *dst,
>  	 *        the folio lock.
>  	 */
>  	if (folio_test_private(src)) {
> -		if (mode == MIGRATE_SYNC)
> +		if (ctl->mode == MIGRATE_SYNC)
>  			nfs_wb_folio(src->mapping->host, src);
>  		if (folio_test_private(src))
>  			return -EBUSY;
>  	}
>  
>  	if (folio_test_private_2(src)) { /* [DEPRECATED] */
> -		if (mode == MIGRATE_ASYNC)
> +		if (ctl->mode == MIGRATE_ASYNC)
>  			return -EBUSY;
>  		folio_wait_private_2(src);
>  	}
>  
> -	return migrate_folio(mapping, dst, src, mode);
> +	return migrate_folio(mapping, dst, src, ctl);
>  }
>  #endif
>  
> diff --git a/include/linux/buffer_head.h b/include/linux/buffer_head.h
> index fd2c7115c054..e4476596c9b6 100644
> --- a/include/linux/buffer_head.h
> +++ b/include/linux/buffer_head.h
> @@ -277,9 +277,11 @@ int block_truncate_page(struct address_space *, loff_t, get_block_t *);
>  
>  #ifdef CONFIG_MIGRATION
>  extern int buffer_migrate_folio(struct address_space *,
> -		struct folio *dst, struct folio *src, enum migrate_mode);
> +		struct folio *dst, struct folio *src,
> +		const struct migrate_control *);
>  extern int buffer_migrate_folio_norefs(struct address_space *,
> -		struct folio *dst, struct folio *src, enum migrate_mode);
> +		struct folio *dst, struct folio *src,
> +		const struct migrate_control *);
>  #else
>  #define buffer_migrate_folio NULL
>  #define buffer_migrate_folio_norefs NULL
> diff --git a/include/linux/fs.h b/include/linux/fs.h
> index f9d1e05e8ae6..93ba6054c478 100644
> --- a/include/linux/fs.h
> +++ b/include/linux/fs.h
> @@ -423,11 +423,11 @@ struct address_space_operations {
>  	void (*free_folio)(struct folio *folio);
>  	ssize_t (*direct_IO)(struct kiocb *, struct iov_iter *iter);
>  	/*
> -	 * migrate the contents of a folio to the specified target. If
> -	 * migrate_mode is MIGRATE_ASYNC, it must not block.
> +	 * Migrate the contents of a folio to the specified target.  The
> +	 * callback must not block when ctl->mode is MIGRATE_ASYNC.
>  	 */
>  	int (*migrate_folio)(struct address_space *, struct folio *dst,
> -			struct folio *src, enum migrate_mode);
> +			struct folio *src, const struct migrate_control *ctl);
>  	int (*launder_folio)(struct folio *);
>  	bool (*is_partially_uptodate) (struct folio *, size_t from,
>  			size_t count);
> diff --git a/include/linux/migrate.h b/include/linux/migrate.h
> index cec5713944e3..123111e7abd3 100644
> --- a/include/linux/migrate.h
> +++ b/include/linux/migrate.h
> @@ -55,7 +55,7 @@ extern const char *migrate_reason_names[MR_TYPES];
>  
>  void putback_movable_pages(struct list_head *l);
>  int migrate_folio(struct address_space *mapping, struct folio *dst,
> -		struct folio *src, enum migrate_mode mode);
> +		struct folio *src, const struct migrate_control *ctl);
>  int migrate_pages(struct list_head *l, new_folio_t new, free_folio_t free,
>  		unsigned long private, const struct migrate_control *ctl,
>  		unsigned int *ret_succeeded);
> diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
> index 0adfa6605653..455d57307f88 100644
> --- a/include/linux/pagemap.h
> +++ b/include/linux/pagemap.h
> @@ -1327,7 +1327,7 @@ bool noop_dirty_folio(struct address_space *mapping, struct folio *folio);
>  
>  #ifdef CONFIG_MIGRATION
>  int filemap_migrate_folio(struct address_space *mapping, struct folio *dst,
> -		struct folio *src, enum migrate_mode mode);
> +		struct folio *src, const struct migrate_control *ctl);
>  #else
>  #define filemap_migrate_folio NULL
>  #endif
> diff --git a/mm/migrate.c b/mm/migrate.c
> index 549f8b57092c..731836664a86 100644
> --- a/mm/migrate.c
> +++ b/mm/migrate.c
> @@ -869,7 +869,7 @@ EXPORT_SYMBOL(folio_migrate_flags);
>  
>  static int __migrate_folio(struct address_space *mapping, struct folio *dst,
>  			   struct folio *src, void *src_private,
> -			   enum migrate_mode mode)
> +			   const struct migrate_control *ctl)
>  {
>  	int rc, expected_count = folio_expected_ref_count(src) + 1;
>  
> @@ -897,7 +897,7 @@ static int __migrate_folio(struct address_space *mapping, struct folio *dst,
>   * @mapping: The address_space containing the folio.
>   * @dst: The folio to migrate the data to.
>   * @src: The folio containing the current data.
> - * @mode: How to migrate the folio.
> + * @ctl: Migration policy for this folio.
>   *
>   * Common logic to directly migrate a single LRU folio suitable for
>   * folios that do not have private data.
> @@ -905,10 +905,10 @@ static int __migrate_folio(struct address_space *mapping, struct folio *dst,
>   * Folios are locked upon entry and exit.
>   */
>  int migrate_folio(struct address_space *mapping, struct folio *dst,
> -		  struct folio *src, enum migrate_mode mode)
> +		  struct folio *src, const struct migrate_control *ctl)
>  {
>  	BUG_ON(folio_test_writeback(src));	/* Writeback must be complete */
> -	return __migrate_folio(mapping, dst, src, NULL, mode);
> +	return __migrate_folio(mapping, dst, src, NULL, ctl);
>  }
>  EXPORT_SYMBOL(migrate_folio);
>  
> @@ -947,8 +947,8 @@ static bool buffer_migrate_lock_buffers(struct buffer_head *head,
>  }
>  
>  static int __buffer_migrate_folio(struct address_space *mapping,
> -		struct folio *dst, struct folio *src, enum migrate_mode mode,
> -		bool check_refs)
> +		struct folio *dst, struct folio *src,
> +		const struct migrate_control *ctl, bool check_refs)
>  {
>  	struct buffer_head *bh, *head;
>  	int rc;
> @@ -956,14 +956,14 @@ static int __buffer_migrate_folio(struct address_space *mapping,
>  
>  	head = folio_buffers(src);
>  	if (!head)
> -		return migrate_folio(mapping, dst, src, mode);
> +		return migrate_folio(mapping, dst, src, ctl);
>  
>  	/* Check whether page does not have extra refs before we do more work */
>  	expected_count = folio_expected_ref_count(src) + 1;
>  	if (folio_ref_count(src) != expected_count)
>  		return -EAGAIN;
>  
> -	if (!buffer_migrate_lock_buffers(head, mode))
> +	if (!buffer_migrate_lock_buffers(head, ctl->mode))
>  		return -EAGAIN;
>  
>  	if (check_refs) {
> @@ -995,7 +995,7 @@ static int __buffer_migrate_folio(struct address_space *mapping,
>  		}
>  	}
>  
> -	rc = filemap_migrate_folio(mapping, dst, src, mode);
> +	rc = filemap_migrate_folio(mapping, dst, src, ctl);
>  	if (rc)
>  		goto unlock_buffers;
>  
> @@ -1022,7 +1022,7 @@ static int __buffer_migrate_folio(struct address_space *mapping,
>   * @mapping: The address space containing @src.
>   * @dst: The folio to migrate to.
>   * @src: The folio to migrate from.
> - * @mode: How to migrate the folio.
> + * @ctl: Migration policy for this folio.
>   *
>   * This function can only be used if the underlying filesystem guarantees
>   * that no other references to @src exist. For example attached buffer
> @@ -1033,9 +1033,10 @@ static int __buffer_migrate_folio(struct address_space *mapping,
>   * Return: 0 on success or a negative errno on failure.
>   */
>  int buffer_migrate_folio(struct address_space *mapping,
> -		struct folio *dst, struct folio *src, enum migrate_mode mode)
> +		struct folio *dst, struct folio *src,
> +		const struct migrate_control *ctl)
>  {
> -	return __buffer_migrate_folio(mapping, dst, src, mode, false);
> +	return __buffer_migrate_folio(mapping, dst, src, ctl, false);
>  }
>  EXPORT_SYMBOL(buffer_migrate_folio);
>  
> @@ -1044,7 +1045,7 @@ EXPORT_SYMBOL(buffer_migrate_folio);
>   * @mapping: The address space containing @src.
>   * @dst: The folio to migrate to.
>   * @src: The folio to migrate from.
> - * @mode: How to migrate the folio.
> + * @ctl: Migration policy for this folio.
>   *
>   * Like buffer_migrate_folio() except that this variant is more careful
>   * and checks that there are also no buffer head references. This function
> @@ -1054,17 +1055,19 @@ EXPORT_SYMBOL(buffer_migrate_folio);
>   * Return: 0 on success or a negative errno on failure.
>   */
>  int buffer_migrate_folio_norefs(struct address_space *mapping,
> -		struct folio *dst, struct folio *src, enum migrate_mode mode)
> +		struct folio *dst, struct folio *src,
> +		const struct migrate_control *ctl)
>  {
> -	return __buffer_migrate_folio(mapping, dst, src, mode, true);
> +	return __buffer_migrate_folio(mapping, dst, src, ctl, true);
>  }
>  EXPORT_SYMBOL_GPL(buffer_migrate_folio_norefs);
>  #endif /* CONFIG_BUFFER_HEAD */
>  
>  int filemap_migrate_folio(struct address_space *mapping,
> -		struct folio *dst, struct folio *src, enum migrate_mode mode)
> +		struct folio *dst, struct folio *src,
> +		const struct migrate_control *ctl)
>  {
> -	return __migrate_folio(mapping, dst, src, folio_get_private(src), mode);
> +	return __migrate_folio(mapping, dst, src, folio_get_private(src), ctl);
>  }
>  EXPORT_SYMBOL_GPL(filemap_migrate_folio);
>  
> @@ -1072,7 +1075,8 @@ EXPORT_SYMBOL_GPL(filemap_migrate_folio);
>   * Default handling if a filesystem does not provide a migration function.
>   */
>  static int fallback_migrate_folio(struct address_space *mapping,
> -		struct folio *dst, struct folio *src, enum migrate_mode mode)
> +		struct folio *dst, struct folio *src,
> +		const struct migrate_control *ctl)
>  {
>  	WARN_ONCE(mapping->a_ops->writepages,
>  			"%ps does not implement migrate_folio\n",
> @@ -1085,9 +1089,9 @@ static int fallback_migrate_folio(struct address_space *mapping,
>  	 * can't migrate automatically.
>  	 */
>  	if (!filemap_release_folio(src, GFP_KERNEL))
> -		return mode == MIGRATE_SYNC ? -EAGAIN : -EBUSY;
> +		return ctl->mode == MIGRATE_SYNC ? -EAGAIN : -EBUSY;
>  
> -	return migrate_folio(mapping, dst, src, mode);
> +	return migrate_folio(mapping, dst, src, ctl);
>  }
>  
>  /*
> @@ -1112,7 +1116,7 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,
>  	VM_BUG_ON_FOLIO(!folio_test_locked(dst), dst);
>  
>  	if (!mapping)
> -		rc = migrate_folio(mapping, dst, src, ctl->mode);
> +		rc = migrate_folio(mapping, dst, src, ctl);
>  	else if (mapping_inaccessible(mapping))
>  		rc = -EOPNOTSUPP;
>  	else if (mapping->a_ops->migrate_folio)
> @@ -1123,9 +1127,9 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,
>  		 * migrate_folio callback. This is the most common path
>  		 * for page migration.
>  		 */
> -		rc = mapping->a_ops->migrate_folio(mapping, dst, src, ctl->mode);
> +		rc = mapping->a_ops->migrate_folio(mapping, dst, src, ctl);
>  	else
> -		rc = fallback_migrate_folio(mapping, dst, src, ctl->mode);
> +		rc = fallback_migrate_folio(mapping, dst, src, ctl);
>  
>  	if (!rc) {
>  		/*
> diff --git a/mm/secretmem.c b/mm/secretmem.c
> index d29865075b6e..5c81833981d2 100644
> --- a/mm/secretmem.c
> +++ b/mm/secretmem.c
> @@ -144,7 +144,8 @@ static const struct file_operations secretmem_fops = {
>  };
>  
>  static int secretmem_migrate_folio(struct address_space *mapping,
> -		struct folio *dst, struct folio *src, enum migrate_mode mode)
> +		struct folio *dst, struct folio *src,
> +		const struct migrate_control *ctl)
>  {
>  	return -EBUSY;
>  }
> diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
> index 625e62e1a031..0a9af8c4768d 100644
> --- a/virt/kvm/guest_memfd.c
> +++ b/virt/kvm/guest_memfd.c
> @@ -457,7 +457,7 @@ static struct file_operations kvm_gmem_fops = {
>  
>  static int kvm_gmem_migrate_folio(struct address_space *mapping,
>  				  struct folio *dst, struct folio *src,
> -				  enum migrate_mode mode)
> +				  const struct migrate_control *ctl)
>  {
>  	WARN_ON_ONCE(1);
>  	return -EINVAL;
> 
> -- 
> 2.43.0
> 
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR