[PATCH RFC v2 0/4] KVM: guest_memfd: folio migration for non-confidential VMs

[PATCH RFC v2 0/4] KVM: guest_memfd: folio migration for non-confidential VMs

[Shivank Garg]

guest_memfd folios are currently always marked unmovable, so the kernel cannot
perform memory compaction, offlining, etc. This is unavoidable for
confidential VMs (SEV-SNP, TDX), since memory is encrypted and copying it
needs firmware assistance. However, for non-confidential VMs (like
Firecracker), we can migrate the folios.

This series enables folio migration for non-confidential guest_memfd and
also lays the groundwork for migrating confidential guest_memfd later.
Once firmware-assisted copying support is available, those VMs can be
made movable, the confidential folio content can be copied separately,
and the destination folio marked with FOLIO_CONTENT_COPIED so
__migrate_folio() skips the host-side folio_mc_copy().

Testing
-------
Host: 7.2-rc5+(62cc902415) + this, 2 NUMA nodes

- KVM selftest: allocate folios on node 0, migrate them to node 1 and
  back and verify resulting NUMA node and the folio contents at each
  step.

- Firecracker [1]: booted a microVM backed by guest_memfd. While the
  guest was running, forced host-side migration of its folios via
  migratepages(8) and explicit move_pages(2) of guest_memfd
  pages. Verify with /proc/firecracker_pid/numa_maps.

Notes
-----
- Sashiko pointed out a pre-existing ABBA deadlock between
  kvm_gmem_error_folio() and truncation. It's being addressed separately
  by Hao Zhang. [2][3]
- Sashiko also asked whether selftest should dynamically discover available
  NUMA nodes instead of hardcoding nodes 0 and 1? This feels like
  over-complication for an artificial scenario. [3] So avoiding it.

[1] https://github.com/firecracker-microvm/firecracker/tree/feature/secret-hiding
    In builder.rs, add GUEST_MEMFD_FLAG_MIGRATABLE to bit-2 and pass it instead
    of GUEST_MEMFD_FLAG_NO_DIRECT_MAP to vm.create_guest_memfd().
[2] https://lore.kernel.org/all/ambEdSPjerZIVN0b@192.168.1.215/
[3] https://sashiko.dev/#/patchset/20260611-shivank-gmem-migrate-v1-0-2d266bfc6f95%40amd.com

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
Changes in v2:
- Make folio migration opt-in through GUEST_MEMFD_FLAG_MIGRATABLE,
  preserving unmovable behavior if userspace don't explictly ask. (Alexandru, David, Sean)
- Add kvm_arch_supports_gmem_migration() so arch can control whether
  GUEST_MEMFD_FLAG_MIGRATABLE is advertised.
- Allocate movable folios with GFP_HIGHUSER_MOVABLE. (David)
- Keep guest_memfd unevictable. (David, Sashiko, Sean)
- Split migrate_folio() implementation and enablement as separate patches.
- Update selftest with new flag.
- Link to v1: https://lore.kernel.org/r/20260611-shivank-gmem-migrate-v1-0-2d266bfc6f95@amd.com

---
Shivank Garg (4):
      mm: split AS_UNMOVABLE back out of AS_INACCESSIBLE
      KVM: guest_memfd: implement folio migration for non-confidential VMs
      KVM: guest_memfd: add GUEST_MEMFD_FLAG_MIGRATABLE
      KVM: selftests: exercise guest_memfd folio migration

 arch/x86/kvm/x86.c                             |  9 +++
 include/linux/kvm_host.h                       |  4 ++
 include/linux/pagemap.h                        | 24 ++++++--
 include/uapi/linux/kvm.h                       |  1 +
 mm/compaction.c                                | 12 ++--
 mm/migrate.c                                   |  2 +-
 tools/include/uapi/linux/kvm.h                 |  1 +
 tools/testing/selftests/kvm/guest_memfd_test.c | 82 ++++++++++++++++++++++++++
 virt/kvm/guest_memfd.c                         | 53 +++++++++++++++--
 9 files changed, 172 insertions(+), 16 deletions(-)
---
base-commit: 62cc90241548d5570ee68e01aaba6506964e9811
change-id: 20260611-shivank-gmem-migrate-8c1c519b30a6

Best regards,
-- 
Shivank Garg <shivankg@amd.com>

[PATCH RFC v2 1/4] mm: split AS_UNMOVABLE back out of AS_INACCESSIBLE

[Shivank Garg]

Commit 27e6a24a4cf3 ("mm, virt: merge AS_UNMOVABLE and AS_INACCESSIBLE")
folded the two flags into one, on the grounds that guest_memfd was the
only user and always set both. But the two flags were added for
different reasons and guard different things:

  AS_UNMOVABLE (0003e2a41468) marks a mapping whose folios cannot be
  migrated.

  AS_INACCESSIBLE (c72ceafbd12c) marks a mapping whose contents must
  not be directly R/W accessed. Its only job is to stop
  truncate_inode_partial_folio() from zeroing the folio.

The merge assumed unmovable and inaccessible were the same thing.
This cannot express a mapping that is inaccessible yet still movable,
which is exactly what guest_memfd wants.

Reintroduce AS_UNMOVABLE and restore the original split: truncate keeps
checking AS_INACCESSIBLE, while migration and compaction go back to
checking AS_UNMOVABLE.

Currently guest_memfd sets both, so the resulting flags and behaviour
are unchanged. Preparatory change to support folio migration for
non-confidential guest_memfd VMs.

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 include/linux/pagemap.h | 24 ++++++++++++++++++++----
 mm/compaction.c         | 12 ++++++------
 mm/migrate.c            |  2 +-
 virt/kvm/guest_memfd.c  |  1 +
 4 files changed, 28 insertions(+), 11 deletions(-)

diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 2c3718d592d6..a7dcaa66e4e3 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -210,6 +210,7 @@ enum mapping_flags {
 	AS_WRITEBACK_MAY_DEADLOCK_ON_RECLAIM = 9,
 	AS_KERNEL_FILE = 10,	/* mapping for a fake kernel file that shouldn't
 				   account usage to user cgroups */
+	AS_UNMOVABLE = 11,	/* The mapping cannot be moved, ever */
 	/* Bits 16-25 are used for FOLIO_ORDER */
 	AS_FOLIO_ORDER_BITS = 5,
 	AS_FOLIO_ORDER_MIN = 16,
@@ -322,11 +323,10 @@ static inline void mapping_clear_stable_writes(struct address_space *mapping)
 static inline void mapping_set_inaccessible(struct address_space *mapping)
 {
 	/*
-	 * It's expected inaccessible mappings are also unevictable. Compaction
-	 * migrate scanner (isolate_migratepages_block()) relies on this to
-	 * reduce page locking.
+	 * The mapping's contents must not be accessed by the CPU through
+	 * the kernel direct map or other internal paths (e.g. zeroing of
+	 * pages during truncation).
 	 */
-	set_bit(AS_UNEVICTABLE, &mapping->flags);
 	set_bit(AS_INACCESSIBLE, &mapping->flags);
 }
 
@@ -335,6 +335,22 @@ static inline bool mapping_inaccessible(const struct address_space *mapping)
 	return test_bit(AS_INACCESSIBLE, &mapping->flags);
 }
 
+static inline void mapping_set_unmovable(struct address_space *mapping)
+{
+	/*
+	 * It's expected unmovable mappings are also unevictable. Compaction
+	 * migrate scanner (isolate_migratepages_block()) relies on this to
+	 * reduce page locking.
+	 */
+	set_bit(AS_UNEVICTABLE, &mapping->flags);
+	set_bit(AS_UNMOVABLE, &mapping->flags);
+}
+
+static inline bool mapping_unmovable(const struct address_space *mapping)
+{
+	return test_bit(AS_UNMOVABLE, &mapping->flags);
+}
+
 static inline void mapping_set_writeback_may_deadlock_on_reclaim(struct address_space *mapping)
 {
 	set_bit(AS_WRITEBACK_MAY_DEADLOCK_ON_RECLAIM, &mapping->flags);
diff --git a/mm/compaction.c b/mm/compaction.c
index f08765ade014..e6b0fdfaf79d 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -1133,22 +1133,22 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
 		if (((mode & ISOLATE_ASYNC_MIGRATE) && is_dirty) ||
 		    (mapping && is_unevictable)) {
 			bool migrate_dirty = true;
-			bool is_inaccessible;
+			bool is_unmovable;
 
 			/*
 			 * Only folios without mappings or that have
 			 * a ->migrate_folio callback are possible to migrate
 			 * without blocking.
 			 *
-			 * Folios from inaccessible mappings are not migratable.
+			 * Folios from unmovable mappings are not migratable.
 			 *
 			 * However, we can be racing with truncation, which can
 			 * free the mapping that we need to check. Truncation
 			 * holds the folio lock until after the folio is removed
 			 * from the page so holding it ourselves is sufficient.
 			 *
-			 * To avoid locking the folio just to check inaccessible,
-			 * assume every inaccessible folio is also unevictable,
+			 * To avoid locking the folio just to check unmovable,
+			 * assume every unmovable folio is also unevictable,
 			 * which is a cheaper test.  If our assumption goes
 			 * wrong, it's not a correctness bug, just potentially
 			 * wasted cycles.
@@ -1161,9 +1161,9 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
 				migrate_dirty = !mapping ||
 						mapping->a_ops->migrate_folio;
 			}
-			is_inaccessible = mapping && mapping_inaccessible(mapping);
+			is_unmovable = mapping && mapping_unmovable(mapping);
 			folio_unlock(folio);
-			if (!migrate_dirty || is_inaccessible)
+			if (!migrate_dirty || is_unmovable)
 				goto isolate_fail_put;
 		}
 
diff --git a/mm/migrate.c b/mm/migrate.c
index d9b23909d716..e48ccd5a9485 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -1100,7 +1100,7 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,
 
 	if (!mapping)
 		rc = migrate_folio(mapping, dst, src, mode);
-	else if (mapping_inaccessible(mapping))
+	else if (mapping_unmovable(mapping))
 		rc = -EOPNOTSUPP;
 	else if (mapping->a_ops->migrate_folio)
 		/*
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index db57c5766ab6..d3b1186b03b3 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -593,6 +593,7 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags)
 	inode->i_size = size;
 	mapping_set_gfp_mask(inode->i_mapping, GFP_HIGHUSER);
 	mapping_set_inaccessible(inode->i_mapping);
+	mapping_set_unmovable(inode->i_mapping);
 	/* Unmovable mappings are supposed to be marked unevictable as well. */
 	WARN_ON_ONCE(!mapping_unevictable(inode->i_mapping));
 

-- 
2.43.0

[PATCH RFC v2 2/4] KVM: guest_memfd: implement folio migration for non-confidential VMs

[Shivank Garg]

Implement kvm_gmem_migrate_folio() using filemap_migrate_folio() for
non-confidential VMs but don't enable it yet. guest_memfd mappings
are still marked unmovable, so MM never calls it.

Migration invokes ->migrate_folio() callback with  folios already
locked and acquires the invalidate lock afterwards. While
kvm_gmem_punch_hole() acquires the invalidate lock and takes folio
lock afterwards during truncation. This could invert the lock order
and cause an ABBA-deadlock situation. Use
filemap_invalidate_trylock_shared() to avoid this.

A subsequent patch adds an opt-in flag to change the mapping to
movable and enable migration. Non-confidential VMs can use
host-side copy via folio_mc_copy(), whereas confidential VMs will
require firmware-assisted copying before they can opt in.

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 virt/kvm/guest_memfd.c | 36 ++++++++++++++++++++++++++++++++++--
 1 file changed, 34 insertions(+), 2 deletions(-)

diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index d3b1186b03b3..ec881cabe8e3 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -488,13 +488,45 @@ static struct file_operations kvm_gmem_fops = {
 	.fallocate	= kvm_gmem_fallocate,
 };
 
+#ifdef CONFIG_MIGRATION
 static int kvm_gmem_migrate_folio(struct address_space *mapping,
 				  struct folio *dst, struct folio *src,
 				  enum migrate_mode mode)
 {
-	WARN_ON_ONCE(1);
-	return -EINVAL;
+	struct inode *inode = mapping->host;
+	pgoff_t start, end;
+	int ret;
+
+	/*
+	 * Migration invokes ->migrate_folio() while holding the folio lock.
+	 * Use a non-blocking trylock to avoid inverting the lock order with
+	 * truncation, which takes the invalidate lock before locking the
+	 * folios.
+	 */
+	if (!filemap_invalidate_trylock_shared(mapping))
+		return -EAGAIN;
+
+	start = src->index;
+	end = start + folio_nr_pages(src);
+
+	kvm_gmem_invalidate_start(inode, start, end);
+
+	/*
+	 * For non-confidential guests the folio is host-readable, so
+	 * filemap_migrate_folio() can copy the contents itself via
+	 * folio_mc_copy().
+	 * For confidential guests, this would need firmware assistance.
+	 */
+	ret = filemap_migrate_folio(mapping, dst, src, mode);
+
+	kvm_gmem_invalidate_end(inode, start, end);
+
+	filemap_invalidate_unlock_shared(mapping);
+	return ret;
 }
+#else
+#define kvm_gmem_migrate_folio NULL
+#endif
 
 static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *folio)
 {

-- 
2.43.0

[PATCH RFC v2 3/4] KVM: guest_memfd: add GUEST_MEMFD_FLAG_MIGRATABLE

[Shivank Garg]

guest_memfd folios are currently always marked unmmovable, which
prevents the MM from invoking ->migrate_folio callback in scenarios
like compaction, offlining, etc. This is unavoidable for confidential VMs
(SEV-SNP, TDX), since memory is encrypted and copying it needs firmware
assistance. However, for non-confidential VMs (like firecracker), we
can still migrate the folios.

Add kvm_arch_supports_gmem_migration() so an architecture can control
whether migration is supported. Add GUEST_MEMFD_FLAG_MIGRATABLE so
userspace can opt-in to folio migration.

For opted-in guest_memfds, use GFP_HIGHUSER_MOVABLE so their folios can
be allocated from ZONE_MOVABLE/CMA and grouped by mobility, otherwise
keep the existing unmmovable behavior.

Remove redundant WARN_ON_ONCE for checking unevictable mapping.

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 arch/x86/kvm/x86.c             |  9 +++++++++
 include/linux/kvm_host.h       |  4 ++++
 include/uapi/linux/kvm.h       |  1 +
 tools/include/uapi/linux/kvm.h |  1 +
 virt/kvm/guest_memfd.c         | 18 ++++++++++++++----
 5 files changed, 29 insertions(+), 4 deletions(-)

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index afcac1042947..4a7016013271 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -14124,6 +14124,15 @@ bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm)
 	return !kvm_arch_has_private_mem(kvm);
 }
 
+/*
+ * Migration of guest_memfd with private memory is not supported yet
+ * as this may require architecture-specific handling.
+ */
+bool kvm_arch_supports_gmem_migration(struct kvm *kvm)
+{
+	return !kvm_arch_has_private_mem(kvm);
+}
+
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
 int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_order)
 {
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index ab8cfaec82d3..7b2f15b21970 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -731,6 +731,7 @@ static inline bool kvm_arch_has_private_mem(struct kvm *kvm)
 
 #ifdef CONFIG_KVM_GUEST_MEMFD
 bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm);
+bool kvm_arch_supports_gmem_migration(struct kvm *kvm);
 
 static inline u64 kvm_gmem_get_supported_flags(struct kvm *kvm)
 {
@@ -738,6 +739,9 @@ static inline u64 kvm_gmem_get_supported_flags(struct kvm *kvm)
 
 	if (!kvm || kvm_arch_supports_gmem_init_shared(kvm))
 		flags |= GUEST_MEMFD_FLAG_INIT_SHARED;
+	if (IS_ENABLED(CONFIG_MIGRATION) &&
+	    (!kvm || kvm_arch_supports_gmem_migration(kvm)))
+		flags |= GUEST_MEMFD_FLAG_MIGRATABLE;
 
 	return flags;
 }
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 419011097fa8..e1615a62ed78 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -1654,6 +1654,7 @@ struct kvm_memory_attributes {
 #define KVM_CREATE_GUEST_MEMFD	_IOWR(KVMIO,  0xd4, struct kvm_create_guest_memfd)
 #define GUEST_MEMFD_FLAG_MMAP		(1ULL << 0)
 #define GUEST_MEMFD_FLAG_INIT_SHARED	(1ULL << 1)
+#define GUEST_MEMFD_FLAG_MIGRATABLE	(1ULL << 2)
 
 struct kvm_create_guest_memfd {
 	__u64 size;
diff --git a/tools/include/uapi/linux/kvm.h b/tools/include/uapi/linux/kvm.h
index 419011097fa8..e1615a62ed78 100644
--- a/tools/include/uapi/linux/kvm.h
+++ b/tools/include/uapi/linux/kvm.h
@@ -1654,6 +1654,7 @@ struct kvm_memory_attributes {
 #define KVM_CREATE_GUEST_MEMFD	_IOWR(KVMIO,  0xd4, struct kvm_create_guest_memfd)
 #define GUEST_MEMFD_FLAG_MMAP		(1ULL << 0)
 #define GUEST_MEMFD_FLAG_INIT_SHARED	(1ULL << 1)
+#define GUEST_MEMFD_FLAG_MIGRATABLE	(1ULL << 2)
 
 struct kvm_create_guest_memfd {
 	__u64 size;
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index ec881cabe8e3..8ae455a6b38d 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -589,6 +589,11 @@ bool __weak kvm_arch_supports_gmem_init_shared(struct kvm *kvm)
 	return true;
 }
 
+bool __weak kvm_arch_supports_gmem_migration(struct kvm *kvm)
+{
+	return false;
+}
+
 static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags)
 {
 	static const char *name = "[kvm-gmem]";
@@ -623,11 +628,16 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags)
 	inode->i_mapping->a_ops = &kvm_gmem_aops;
 	inode->i_mode |= S_IFREG;
 	inode->i_size = size;
-	mapping_set_gfp_mask(inode->i_mapping, GFP_HIGHUSER);
 	mapping_set_inaccessible(inode->i_mapping);
-	mapping_set_unmovable(inode->i_mapping);
-	/* Unmovable mappings are supposed to be marked unevictable as well. */
-	WARN_ON_ONCE(!mapping_unevictable(inode->i_mapping));
+	/* guest_memfd mappings should be marked unevictable. */
+	mapping_set_unevictable(inode->i_mapping);
+
+	if (flags & GUEST_MEMFD_FLAG_MIGRATABLE) {
+		mapping_set_gfp_mask(inode->i_mapping, GFP_HIGHUSER_MOVABLE);
+	} else {
+		mapping_set_gfp_mask(inode->i_mapping, GFP_HIGHUSER);
+		mapping_set_unmovable(inode->i_mapping);
+	}
 
 	GMEM_I(inode)->flags = flags;
 

-- 
2.43.0

[PATCH RFC v2 4/4] KVM: selftests: exercise guest_memfd folio migration

[Shivank Garg]

Add a migration test to guest_memfd_test using guest_memfd files created
with MMAP, INIT_SHARED, and MIGRATABLE. MMAP and INIT_SHARED are required
here because move_pages() operates on userspace mapping. Run the test
only when the VM supports all three flags and the system has at least two
NUMA nodes (skipped otherwise).

Migrate every folio from node 0 to node 1 with move_pages(), and
verify both the resulting node and contents. Migrate the folios back
and verify their contents again.

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 tools/testing/selftests/kvm/guest_memfd_test.c | 82 ++++++++++++++++++++++++++
 1 file changed, 82 insertions(+)

diff --git a/tools/testing/selftests/kvm/guest_memfd_test.c b/tools/testing/selftests/kvm/guest_memfd_test.c
index 2233d871a38f..77e034a9d587 100644
--- a/tools/testing/selftests/kvm/guest_memfd_test.c
+++ b/tools/testing/selftests/kvm/guest_memfd_test.c
@@ -171,6 +171,82 @@ static void test_numa_allocation(int fd, size_t total_size)
 	kvm_munmap(mem, total_size);
 }
 
+/*
+ * Each page is filled with a distinct byte (its index). Check every byte that
+ * data is intact after migration.
+ */
+static void verify_page(const char *page, int page_idx, size_t size,
+			const char *when)
+{
+	char expected = (char)(page_idx & 0xff);
+	size_t off;
+
+	for (off = 0; off < size; off++)
+		TEST_ASSERT(page[off] == expected,
+			    "Page %d corrupted at offset %zu %s", page_idx, off, when);
+}
+
+static void test_migrate_folio(int fd, size_t total_size)
+{
+	const unsigned long nodemask_0 = 1; /* nid: 0 */
+	unsigned long maxnode = BITS_PER_TYPE(nodemask_0);
+	int page_count = total_size / page_size;
+	int *status, *nodes;
+	void **addr;
+	char *mem;
+	int i;
+
+	if (!is_multi_numa_node_system())
+		return;
+
+	mem = kvm_mmap(total_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd);
+
+	addr = calloc(page_count, sizeof(*addr));
+	status = calloc(page_count, sizeof(*status));
+	nodes = calloc(page_count, sizeof(*nodes));
+	TEST_ASSERT(addr && status && nodes, "Failed to allocate page arrays");
+
+	/* Allocate all folios on node 0 and fill each with a known pattern. */
+	kvm_mbind(mem, total_size, MPOL_BIND, &nodemask_0, maxnode, 0);
+	for (i = 0; i < page_count; i++) {
+		memset(mem + i * page_size, (char)(i & 0xff), page_size);
+		addr[i] = mem + i * page_size;
+	}
+
+	kvm_move_pages(0, page_count, addr, NULL, status, 0);
+	for (i = 0; i < page_count; i++)
+		TEST_ASSERT(status[i] == 0, "Page %d should be on node 0", i);
+
+	/* Migrate node 0 -> 1, then check both the location and the data. */
+	for (i = 0; i < page_count; i++)
+		nodes[i] = 1;
+	kvm_move_pages(0, page_count, addr, nodes, status, MPOL_MF_MOVE);
+
+	kvm_move_pages(0, page_count, addr, NULL, status, 0);
+	for (i = 0; i < page_count; i++)
+		TEST_ASSERT(status[i] == 1,
+			    "Page %d should be on node 1 after migration", i);
+	for (i = 0; i < page_count; i++)
+		verify_page(mem + i * page_size, i, page_size, "after migration");
+
+	/* Migrate back node 1 -> 0, then re-check the location and the data. */
+	for (i = 0; i < page_count; i++)
+		nodes[i] = 0;
+	kvm_move_pages(0, page_count, addr, nodes, status, MPOL_MF_MOVE);
+
+	kvm_move_pages(0, page_count, addr, NULL, status, 0);
+	for (i = 0; i < page_count; i++)
+		TEST_ASSERT(status[i] == 0,
+			    "Page %d should be on node 0 after round-trip", i);
+	for (i = 0; i < page_count; i++)
+		verify_page(mem + i * page_size, i, page_size, "after round-trip");
+
+	free(addr);
+	free(status);
+	free(nodes);
+	kvm_munmap(mem, total_size);
+}
+
 static void test_collapse(int fd, u64 flags)
 {
 	const size_t pmd_size = get_trans_hugepagesz();
@@ -472,6 +548,9 @@ static void __test_guest_memfd(struct kvm_vm *vm, u64 flags)
 static void test_guest_memfd(unsigned long vm_type)
 {
 	struct kvm_vm *vm = vm_create_barebones_type(vm_type);
+	const u64 migrate_flags = GUEST_MEMFD_FLAG_MMAP |
+				  GUEST_MEMFD_FLAG_INIT_SHARED |
+				  GUEST_MEMFD_FLAG_MIGRATABLE;
 	u64 flags;
 
 	test_guest_memfd_flags(vm);
@@ -486,6 +565,9 @@ static void test_guest_memfd(unsigned long vm_type)
 	if (flags & GUEST_MEMFD_FLAG_INIT_SHARED)
 		__test_guest_memfd(vm, GUEST_MEMFD_FLAG_MMAP |
 				       GUEST_MEMFD_FLAG_INIT_SHARED);
+	/* Testing with move_pages() requires MMAP and INIT_SHARED. */
+	if ((flags & migrate_flags) == migrate_flags)
+		gmem_test(migrate_folio, vm, migrate_flags);
 
 	kvm_vm_free(vm);
 }

-- 
2.43.0

Re: [PATCH RFC v2 4/4] KVM: selftests: exercise guest_memfd folio migration

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Hardcoding of NUMA node IDs 0 and 1 makes the test brittle and prone to false failures on systems with sparse, non-contiguous, or memoryless nodes.
--

--- Patch [4]: [PATCH RFC v2 4/4] KVM: selftests: exercise guest_memfd folio migration ---
commit 3681825fc67afc2ab3cf2b50c6cec1297cae94b5
Author: Shivank Garg <shivankg@amd.com>

KVM: selftests: exercise guest_memfd folio migration

This commit adds a migration test to guest_memfd_test to verify that
guest_memfd folios can be migrated across NUMA nodes using move_pages().
It allocates folios on node 0, migrates them to node 1, verifies the
contents, and then migrates them back.

> diff --git a/tools/testing/selftests/kvm/guest_memfd_test.c b/tools/testing/selftests/kvm/guest_memfd_test.c
> --- a/tools/testing/selftests/kvm/guest_memfd_test.c
> +++ b/tools/testing/selftests/kvm/guest_memfd_test.c
[ ... ]
> +static void test_migrate_folio(int fd, size_t total_size)
> +{
> +	const unsigned long nodemask_0 = 1; /* nid: 0 */
> +	unsigned long maxnode = BITS_PER_TYPE(nodemask_0);
> +	int page_count = total_size / page_size;
> +	int *status, *nodes;
> +	void **addr;
> +	char *mem;
> +	int i;
> +
> +	if (!is_multi_numa_node_system())
> +		return;
> +
> +	mem = kvm_mmap(total_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd);
> +
> +	addr = calloc(page_count, sizeof(*addr));
> +	status = calloc(page_count, sizeof(*status));
> +	nodes = calloc(page_count, sizeof(*nodes));
> +	TEST_ASSERT(addr && status && nodes, "Failed to allocate page arrays");
> +
> +	/* Allocate all folios on node 0 and fill each with a known pattern. */
> +	kvm_mbind(mem, total_size, MPOL_BIND, &nodemask_0, maxnode, 0);
> +	for (i = 0; i < page_count; i++) {
> +		memset(mem + i * page_size, (char)(i & 0xff), page_size);
> +		addr[i] = mem + i * page_size;
> +	}
> +
> +	kvm_move_pages(0, page_count, addr, NULL, status, 0);
> +	for (i = 0; i < page_count; i++)
> +		TEST_ASSERT(status[i] == 0, "Page %d should be on node 0", i);
> +
> +	/* Migrate node 0 -> 1, then check both the location and the data. */
> +	for (i = 0; i < page_count; i++)
> +		nodes[i] = 1;

[Severity: Medium]
Could this hardcoding of NUMA node IDs cause false positive test failures?

If a system has sparse, non-contiguous, or memoryless nodes (for example,
nodes 0 and 2 exist, but node 1 is absent), is_multi_numa_node_system()
will return true, but the initial allocation on node 0 or the migration to
node 1 could fail.

Would it be better to dynamically discover available memory nodes in
test_migrate_folio() instead of assuming nodes 0 and 1 always exist and
have memory available?

> +	kvm_move_pages(0, page_count, addr, nodes, status, MPOL_MF_MOVE);
> +
> +	kvm_move_pages(0, page_count, addr, NULL, status, 0);

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260728-shivank-gmem-migrate-v2-0-269ac1f84e2b@amd.com?part=4

Re: [PATCH RFC v2 2/4] KVM: guest_memfd: implement folio migration for non-confidential VMs

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Pre-existing issues:
- [High] Pre-existing ABBA deadlock between `mapping->invalidate_lock` and `folio_lock` via `kvm_gmem_error_folio`.
- [High] Unpaired `kvm_mmu_invalidate_start` due to lockless `f->bindings` xarray modification during `kvm_gmem_unbind`.
--

--- Patch [2]: [PATCH RFC v2 2/4] KVM: guest_memfd: implement folio migration for non-confidential VMs ---
commit cb30b8cd0e73b5e7ad80d0285306810cbf4d963e
Author: Shivank Garg <shivankg@amd.com>
KVM: guest_memfd: implement folio migration for non-confidential VMs

Implement kvm_gmem_migrate_folio using filemap_migrate_folio for
non-confidential VMs but don't enable it yet. Guest_memfd mappings are still
marked unmovable, so MM never calls it.

> diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
> index d3b1186b03b3a..ec881cabe8e3e 100644
> --- a/virt/kvm/guest_memfd.c
> +++ b/virt/kvm/guest_memfd.c
[ ... ]
> +	start = src->index;
> +	end = start + folio_nr_pages(src);
> +
> +	kvm_gmem_invalidate_start(inode, start, end);
> +
> +	/*
> +	 * For non-confidential guests the folio is host-readable, so
> +	 * filemap_migrate_folio() can copy the contents itself via
> +	 * folio_mc_copy().
> +	 * For confidential guests, this would need firmware assistance.
> +	 */
> +	ret = filemap_migrate_folio(mapping, dst, src, mode);
> +
> +	kvm_gmem_invalidate_end(inode, start, end);

[Severity: High]
This is a pre-existing issue, but does the iteration over bindings here race
with kvm_gmem_unbind?

If a guest_memfd file is being closed and its refcount reaches zero
concurrently, kvm_gmem_unbind takes the !file branch and removes the memslot
from f->bindings without taking filemap_invalidate_lock.

Could this lockless modification happen between the calls to
kvm_gmem_invalidate_start and kvm_gmem_invalidate_end?

If f->bindings is modified during this window, __kvm_gmem_invalidate_end
might fail to find the binding and skip the slot, resulting in an unpaired
start call. This would permanently leak the mmu_invalidate_in_progress
counter and cause guest page faults to hang indefinitely.

> +
> +	filemap_invalidate_unlock_shared(mapping);
> +	return ret;
>  }
> +#else
> +#define kvm_gmem_migrate_folio NULL
> +#endif
>  
>  static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *folio)
>  {

[Severity: High]
This isn't a bug introduced by this patch, but does kvm_gmem_error_folio
have the exact same ABBA deadlock that you correctly avoided above in
kvm_gmem_migrate_folio?

In kvm_gmem_error_folio, it unconditionally calls
filemap_invalidate_lock_shared while the caller (memory_failure) already
holds the folio lock.

Meanwhile, the opposing truncation path in kvm_gmem_punch_hole establishes
the strict lock ordering of acquiring mapping->invalidate_lock followed by
locking the folio.

Could a concurrent fallocate PUNCH_HOLE and a memory failure on the same
guest_memfd file deadlock the kernel here?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260728-shivank-gmem-migrate-v2-0-269ac1f84e2b@amd.com?part=2

Re: [PATCH RFC v2 3/4] KVM: guest_memfd: add GUEST_MEMFD_FLAG_MIGRATABLE

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Pre-existing issues:
- [Critical] Accessing memslots in `__kvm_gmem_invalidate_start` without holding `kvm->srcu` allows a use-after-free if the memslot is concurrently deleted while the guest_memfd is being closed.
- [High] Concurrent execution of `kvm_gmem_unbind` on a dying file and `kvm_gmem_migrate_folio` causes asymmetric `kvm_mmu_invalidate_begin`/`end` pairing, permanently hanging the VM in page faults.
--

--- Patch [3]: [PATCH RFC v2 3/4] KVM: guest_memfd: add GUEST_MEMFD_FLAG_MIGRATABLE ---
commit 5a71d0799db0fa4a13e00aa0d626b30f60552d46
Author: Shivank Garg <shivankg@amd.com>

KVM: guest_memfd: add GUEST_MEMFD_FLAG_MIGRATABLE

This commit introduces GUEST_MEMFD_FLAG_MIGRATABLE and
kvm_arch_supports_gmem_migration() to control migration support. It allocates
opted-in guest_memfds with GFP_HIGHUSER_MOVABLE, allowing folios to be
migrated by the memory manager in non-confidential VMs.

[ ... ]

> diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
> index ec881cabe8e3e..8ae455a6b38df 100644
> --- a/virt/kvm/guest_memfd.c
> +++ b/virt/kvm/guest_memfd.c

[ ... ]

> @@ -623,11 +628,16 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags)
>  	inode->i_mapping->a_ops = &kvm_gmem_aops;
>  	inode->i_mode |= S_IFREG;
>  	inode->i_size = size;
> -	mapping_set_gfp_mask(inode->i_mapping, GFP_HIGHUSER);
>  	mapping_set_inaccessible(inode->i_mapping);
> -	mapping_set_unmovable(inode->i_mapping);
> -	/* Unmovable mappings are supposed to be marked unevictable as well. */
> -	WARN_ON_ONCE(!mapping_unevictable(inode->i_mapping));
> +	/* guest_memfd mappings should be marked unevictable. */
> +	mapping_set_unevictable(inode->i_mapping);
> +
> +	if (flags & GUEST_MEMFD_FLAG_MIGRATABLE) {

[Severity: Critical]
This isn't a bug introduced by this patch, but enabling migration here
exposes a pre-existing use-after-free risk in __kvm_gmem_invalidate_start().

If an unprivileged user creates a migratable guest_memfd and binds it, then
concurrently closes the file descriptor and deletes the KVM memslot, memory
compaction can invoke kvm_gmem_migrate_folio() which calls
__kvm_gmem_invalidate_start().

Because kvm_gmem_unbind() bypasses filemap_invalidate_lock when the file
is dying, it does not wait for migration. __kvm_gmem_invalidate_start() then
dereferences the memslot without holding kvm->srcu:

virt/kvm/guest_memfd.c:__kvm_gmem_invalidate_start() {
    xa_for_each_range(&f->bindings, index, slot, start, end - 1) {
        pgoff_t pgoff = slot->gmem.pgoff;
        struct kvm_gfn_range gfn_range = {
            .start = slot->base_gfn + max(pgoff, start) - pgoff,
        ...
}

Is there a risk that the memslot deletion path completes synchronize_srcu()
and calls kfree(slot) while __kvm_gmem_invalidate_start() is still accessing
it?


[Severity: High]
This is also a pre-existing issue, but exposing the migratable flag makes it
reachable. Could this lead to asymmetric kvm_mmu_invalidate_begin/end pairing
and a permanent VM hang?

When kvm_gmem_unbind() executes concurrently with kvm_gmem_migrate_folio()
on a dying file, it bypasses the lock and removes the binding from the
xarray between migration's invalidate_start and invalidate_end:

virt/kvm/guest_memfd.c:kvm_gmem_unbind() {
    if (!file) {
        __kvm_gmem_unbind(slot, slot->gmem.file->private_data);
        return;
    }
}

When the migration finishes, __kvm_gmem_invalidate_end() conditionally
checks xa_find():

virt/kvm/guest_memfd.c:__kvm_gmem_invalidate_end() {
    if (xa_find(&f->bindings, &start, end - 1, XA_PRESENT)) {
        KVM_MMU_LOCK(kvm);
        kvm_mmu_invalidate_end(kvm);
        KVM_MMU_UNLOCK(kvm);
    }
}

Since the binding disappeared, doesn't this skip the _end notification,
permanently leaking KVM's mmu_invalidate_in_progress counter and causing
subsequent page faults to infinitely loop in mmu_invalidate_retry_gfn()?

> +		mapping_set_gfp_mask(inode->i_mapping, GFP_HIGHUSER_MOVABLE);
> +	} else {
> +		mapping_set_gfp_mask(inode->i_mapping, GFP_HIGHUSER);
> +		mapping_set_unmovable(inode->i_mapping);
> +	}
>  
>  	GMEM_I(inode)->flags = flags;

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260728-shivank-gmem-migrate-v2-0-269ac1f84e2b@amd.com?part=3