[PATCH v3 0/9] KVM: guest_memfd: folio migration for non-confidential VMs

[PATCH v3 0/9] KVM: guest_memfd: folio migration for non-confidential VMs

[Shivank Garg]

guest_memfd folios are currently always marked unmovable, so the kernel cannot
perform memory compaction, offlining, etc. This is unavoidable for
confidential VMs (SEV-SNP, TDX), since memory is encrypted and copying it
needs firmware assistance. However, for non-confidential VMs (like
Firecracker), we can migrate the folios.

This series enables folio migration for non-confidential guest_memfd and
also lays the groundwork for migrating confidential guest_memfd later.
Once firmware-assisted copying support is available, those VMs can be
made movable, the confidential folio content can be copied separately,
and the destination folio marked with FOLIO_CONTENT_COPIED[4] so
__migrate_folio() skips the host-side folio_mc_copy().

Testing
-------
Host: 7.2-rc6+(c21bb419386) + this, AMD EPYC ZEN 3, 2 NUMA nodes

- KVM selftest: allocate folios on node 0, migrate them to node 1 and
  back and verify resulting NUMA node and the folio contents at each
  step.

- Firecracker [1]: booted a microVM backed by guest_memfd. While the
  guest was running, forced host-side migration of its folios via
  migratepages(8) and explicit move_pages(2) of guest_memfd
  pages. Verify with /proc/firecracker_pid/numa_maps.

Notes
-----
- Sashiko pointed out a pre-existing ABBA deadlock between
  kvm_gmem_error_folio() and truncation. It's being addressed separately
  by Hao Zhang. [2][3]

[1] https://github.com/firecracker-microvm/firecracker/tree/feature/secret-hiding
    In builder.rs, add GUEST_MEMFD_FLAG_MIGRATABLE to bit-2 and pass it instead
    of GUEST_MEMFD_FLAG_NO_DIRECT_MAP to vm.create_guest_memfd().
[2] https://lore.kernel.org/all/ambEdSPjerZIVN0b@192.168.1.215/
[3] https://sashiko.dev/#/patchset/20260611-shivank-gmem-migrate-v1-0-2d266bfc6f95%40amd.com
[4] https://lore.kernel.org/all/20260630-shivank-batch-migrate-offload-v6-1-da95d7e8b8a2@amd.com

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
Changes in v3:
- Fix unbalanced mmu_invalidate_in_progress count unbinding dying guest_memfd. (Sashiko)
- Fix maxnode handling in xapic_ipi_test selftest.
- Add GUEST_MEMFD_FLAG_MIGRATABLE documentation
- Replace open-coded sizeof() * 8 calculation with BITS_PER_TYPE()
- Add get_numa_mem_nodes() and use  MPOL_F_MEMS_ALLOWED for allowed NUMA ndoes
  instead of hardcoded NUMA node IDs. (Sashiko)
- Extend migration selftest to verify rejection without MIGRATABLE flag and
  move repeated checks into common helpers.
- Drop RFC tag.
- Link to v2: https://lore.kernel.org/r/20260728-shivank-gmem-migrate-v2-0-269ac1f84e2b@amd.com

Changes in v2:
- Make folio migration opt-in through GUEST_MEMFD_FLAG_MIGRATABLE,
  preserving unmovable behavior if userspace don't explictly ask. (Alexandru, David, Sean)
- Add kvm_arch_supports_gmem_migration() so arch can control whether
  GUEST_MEMFD_FLAG_MIGRATABLE is advertised.
- Allocate movable folios with GFP_HIGHUSER_MOVABLE. (David)
- Keep guest_memfd unevictable. (David, Sashiko, Sean)
- Split migrate_folio() implementation and enablement as separate patches.
- Update selftest with new flag.
- Link to v1: https://lore.kernel.org/r/20260611-shivank-gmem-migrate-v1-0-2d266bfc6f95@amd.com

---
Shivank Garg (9):
      KVM: guest_memfd: take the invalidate lock when unbinding a dying file
      mm: split AS_UNMOVABLE back out of AS_INACCESSIBLE
      KVM: guest_memfd: implement folio migration for non-confidential VMs
      KVM: guest_memfd: add GUEST_MEMFD_FLAG_MIGRATABLE
      KVM: selftests: fix maxnode arguments in xapic_ipi_test
      KVM: selftests: use BITS_PER_TYPE() for NUMA masks
      KVM: selftests: add get_numa_mem_nodes()
      KVM: selftests: use allowed NUMA nodes in guest_memfd_test
      KVM: selftests: exercise guest_memfd folio migration

 Documentation/virt/kvm/api.rst                   |   3 +
 arch/x86/kvm/x86.c                               |   9 ++
 include/linux/kvm_host.h                         |   4 +
 include/linux/pagemap.h                          |  24 ++-
 include/uapi/linux/kvm.h                         |   1 +
 mm/compaction.c                                  |  12 +-
 mm/migrate.c                                     |   2 +-
 tools/include/uapi/linux/kvm.h                   |   1 +
 tools/testing/selftests/kvm/guest_memfd_test.c   | 193 +++++++++++++++++++----
 tools/testing/selftests/kvm/include/numaif.h     |  52 +-----
 tools/testing/selftests/kvm/x86/xapic_ipi_test.c |  14 +-
 virt/kvm/guest_memfd.c                           |  76 +++++++--
 12 files changed, 284 insertions(+), 107 deletions(-)
---
base-commit: c21bb4193868a8de71fc4693fa741e195fdf5d86
change-id: 20260611-shivank-gmem-migrate-8c1c519b30a6

Best regards,
-- 
Shivank Garg <shivankg@amd.com>

[PATCH v3 1/9] KVM: guest_memfd: take the invalidate lock when unbinding a dying file

[Shivank Garg]

kvm_gmem_unbind() skips mapping->invalidate_lock when the guest_memfd
file is already dying. All other paths that modify f->bindings hold
that lock.

kvm_gmem_invalidate_{start,end}() checks f->bindings independently to
decide whether to begin or end KVM MMU invalidations. So, the bindings
must remain stable between the two calls. If a binding is removed in that
window, start increments mmu_invalidate_in_progress but end does not
decrement it. Example, unbind race with memory failure:

  CPU 0: memory failure               CPU 1: memslot delete
  ----------------------------------  ---------------------------
                                      (guest_memfd file is dying)
  kvm_gmem_error_folio()
    kvm_gmem_invalidate_start()
      finds binding
      mmu_invalidate_in_progress++
                                      kvm_gmem_unbind()
                                        get_file_active() fails
                                        store NULL in bindings
    kvm_gmem_invalidate_end()
      no binding found
      counter stays elevated

mmu_invalidate_retry() then returns 1 forever, so guest page faults
retry without ever installing a mapping and the guest hangs.

Take the invalidate lock in the dying-file path too. This prevents unbind
from removing a binding and leaking mmu_invalidate_in_progress. This is
safe because any caller that reaches this path holds slots_lock, so
kvm_gmem_release() cannot nullify the slots->gmem.file, until
kvm_gmem_unbind() finishes.

Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/20260728092027.225CF1F000E9@smtp.kernel.org
Fixes: ae431059e75d ("KVM: guest_memfd: Remove bindings on memslot deletion when gmem is dying")
Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 virt/kvm/guest_memfd.c | 23 ++++++++++++++---------
 1 file changed, 14 insertions(+), 9 deletions(-)

diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index db57c5766ab6..45cbdf4801ec 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -721,6 +721,8 @@ static void __kvm_gmem_unbind(struct kvm_memory_slot *slot, struct gmem_file *f)
 
 void kvm_gmem_unbind(struct kvm_memory_slot *slot)
 {
+	struct file *gmem_file;
+
 	/*
 	 * Nothing to do if the underlying file was _already_ closed, as
 	 * kvm_gmem_release() invalidates and nullifies all bindings.
@@ -733,21 +735,24 @@ void kvm_gmem_unbind(struct kvm_memory_slot *slot)
 	/*
 	 * However, if the file is _being_ closed, then the bindings need to be
 	 * removed as kvm_gmem_release() might not run until after the memslot
-	 * is freed.  Note, modifying the bindings is safe even though the file
-	 * is dying as kvm_gmem_release() nullifies slot->gmem.file under
+	 * is freed.  Note, dereferencing the dying file is safe as
+	 * kvm_gmem_release() nullifies slot->gmem.file under
 	 * slots_lock, and only puts its reference to KVM after destroying all
 	 * bindings.  I.e. reaching this point means kvm_gmem_release() hasn't
 	 * yet destroyed the bindings or freed the gmem_file, and can't do so
 	 * until the caller drops slots_lock.
 	 */
-	if (!file) {
-		__kvm_gmem_unbind(slot, slot->gmem.file->private_data);
-		return;
-	}
+	gmem_file = file ?: slot->gmem.file;
 
-	filemap_invalidate_lock(file->f_mapping);
-	__kvm_gmem_unbind(slot, file->private_data);
-	filemap_invalidate_unlock(file->f_mapping);
+	/*
+	 * Take the invalidate lock even for a dying file.  Otherwise,
+	 * kvm_gmem_invalidate_start() can find the binding and increment
+	 * mmu_invalidate_in_progress while kvm_gmem_invalidate_end() misses
+	 * the removed binding and skips decrement.
+	 */
+	filemap_invalidate_lock(gmem_file->f_mapping);
+	__kvm_gmem_unbind(slot, gmem_file->private_data);
+	filemap_invalidate_unlock(gmem_file->f_mapping);
 }
 
 /* Returns a locked folio on success.  */

-- 
2.43.0

Re: [PATCH v3 1/9] KVM: guest_memfd: take the invalidate lock when unbinding a dying file

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [Critical] kvm_gmem_bind() ignores the return value of xa_store_range(), which can fail with -ENOMEM, leading to inconsistent memslot tracking and guest-to-host use-after-free.
--

--- Patch [1]: [PATCH v3 1/9] KVM: guest_memfd: take the invalidate lock when unbinding a dying file ---
commit 4192ee62a3c9cd53bedefe498647d781d59dfdf5
Author: Shivank Garg <shivankg@amd.com>
KVM: guest_memfd: take the invalidate lock when unbinding a dying file

This patch takes the invalidate lock during the dying-file path in
kvm_gmem_unbind() to prevent a race condition with memory failures.
It ensures the mmu_invalidate_in_progress counter remains balanced
when a binding is removed concurrently.

Link: https://lore.kernel.org/all/20260728092027.225CF1F000E9@smtp.kernel.org

> diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
> index db57c5766ab60..45cbdf4801ece 100644
> --- a/virt/kvm/guest_memfd.c
> +++ b/virt/kvm/guest_memfd.c

[ ... ]

[Severity: Critical]
This isn't a bug introduced by this patch, but there is a pre-existing
issue in kvm_gmem_bind() where an error return is ignored:

virt/kvm/guest_memfd.c:kvm_gmem_bind() {
        ...
        xa_store_range(&f->bindings, start, end - 1, slot, GFP_KERNEL);
        filemap_invalidate_unlock(inode->i_mapping);

        /*
         * Drop the reference to the file, even on success.  The file pins KVM,
         * not the other way 'round.  Active bindings are invalidated if the
         * file is closed before memslots are destroyed.
         */
        r = 0;
err:
        fput(file);
        return r;
}

Does this code ignore the return value of xa_store_range(), which can fail
with -ENOMEM under memory pressure?

If xa_store_range() fails, the memslot won't be recorded in the guest_memfd
file's bindings. Subsequent host invalidations, such as FALLOC_FL_PUNCH_HOLE
or a memory failure, would fail to find the memslot and wouldn't invalidate
the guest's page tables.

Can this allow the guest to retain access to freed host memory, leading to
host memory use-after-free or cross-VM data corruption if the memory is
reallocated to other host processes?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260805-shivank-gmem-migrate-v3-0-00d8bdec4e1d@amd.com?part=1

[PATCH v3 2/9] mm: split AS_UNMOVABLE back out of AS_INACCESSIBLE

[Shivank Garg]

Commit 27e6a24a4cf3 ("mm, virt: merge AS_UNMOVABLE and AS_INACCESSIBLE")
folded the two flags into one, on the grounds that guest_memfd was the
only user and always set both. But the two flags were added for
different reasons and guard different things:

  AS_UNMOVABLE (0003e2a41468) marks a mapping whose folios cannot be
  migrated.

  AS_INACCESSIBLE (c72ceafbd12c) marks a mapping whose contents must
  not be directly R/W accessed. Its only job is to stop
  truncate_inode_partial_folio() from zeroing the folio.

The merge assumed unmovable and inaccessible were the same thing.
This cannot express a mapping that is inaccessible yet still movable,
which is exactly what guest_memfd wants.

Reintroduce AS_UNMOVABLE and restore the original split: truncate keeps
checking AS_INACCESSIBLE, while migration and compaction go back to
checking AS_UNMOVABLE.

Currently guest_memfd sets both, so the resulting flags and behaviour
are unchanged. Preparatory change to support folio migration for
non-confidential guest_memfd VMs.

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 include/linux/pagemap.h | 24 ++++++++++++++++++++----
 mm/compaction.c         | 12 ++++++------
 mm/migrate.c            |  2 +-
 virt/kvm/guest_memfd.c  |  1 +
 4 files changed, 28 insertions(+), 11 deletions(-)

diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 2c3718d592d6..a7dcaa66e4e3 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -210,6 +210,7 @@ enum mapping_flags {
 	AS_WRITEBACK_MAY_DEADLOCK_ON_RECLAIM = 9,
 	AS_KERNEL_FILE = 10,	/* mapping for a fake kernel file that shouldn't
 				   account usage to user cgroups */
+	AS_UNMOVABLE = 11,	/* The mapping cannot be moved, ever */
 	/* Bits 16-25 are used for FOLIO_ORDER */
 	AS_FOLIO_ORDER_BITS = 5,
 	AS_FOLIO_ORDER_MIN = 16,
@@ -322,11 +323,10 @@ static inline void mapping_clear_stable_writes(struct address_space *mapping)
 static inline void mapping_set_inaccessible(struct address_space *mapping)
 {
 	/*
-	 * It's expected inaccessible mappings are also unevictable. Compaction
-	 * migrate scanner (isolate_migratepages_block()) relies on this to
-	 * reduce page locking.
+	 * The mapping's contents must not be accessed by the CPU through
+	 * the kernel direct map or other internal paths (e.g. zeroing of
+	 * pages during truncation).
 	 */
-	set_bit(AS_UNEVICTABLE, &mapping->flags);
 	set_bit(AS_INACCESSIBLE, &mapping->flags);
 }
 
@@ -335,6 +335,22 @@ static inline bool mapping_inaccessible(const struct address_space *mapping)
 	return test_bit(AS_INACCESSIBLE, &mapping->flags);
 }
 
+static inline void mapping_set_unmovable(struct address_space *mapping)
+{
+	/*
+	 * It's expected unmovable mappings are also unevictable. Compaction
+	 * migrate scanner (isolate_migratepages_block()) relies on this to
+	 * reduce page locking.
+	 */
+	set_bit(AS_UNEVICTABLE, &mapping->flags);
+	set_bit(AS_UNMOVABLE, &mapping->flags);
+}
+
+static inline bool mapping_unmovable(const struct address_space *mapping)
+{
+	return test_bit(AS_UNMOVABLE, &mapping->flags);
+}
+
 static inline void mapping_set_writeback_may_deadlock_on_reclaim(struct address_space *mapping)
 {
 	set_bit(AS_WRITEBACK_MAY_DEADLOCK_ON_RECLAIM, &mapping->flags);
diff --git a/mm/compaction.c b/mm/compaction.c
index f08765ade014..e6b0fdfaf79d 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -1133,22 +1133,22 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
 		if (((mode & ISOLATE_ASYNC_MIGRATE) && is_dirty) ||
 		    (mapping && is_unevictable)) {
 			bool migrate_dirty = true;
-			bool is_inaccessible;
+			bool is_unmovable;
 
 			/*
 			 * Only folios without mappings or that have
 			 * a ->migrate_folio callback are possible to migrate
 			 * without blocking.
 			 *
-			 * Folios from inaccessible mappings are not migratable.
+			 * Folios from unmovable mappings are not migratable.
 			 *
 			 * However, we can be racing with truncation, which can
 			 * free the mapping that we need to check. Truncation
 			 * holds the folio lock until after the folio is removed
 			 * from the page so holding it ourselves is sufficient.
 			 *
-			 * To avoid locking the folio just to check inaccessible,
-			 * assume every inaccessible folio is also unevictable,
+			 * To avoid locking the folio just to check unmovable,
+			 * assume every unmovable folio is also unevictable,
 			 * which is a cheaper test.  If our assumption goes
 			 * wrong, it's not a correctness bug, just potentially
 			 * wasted cycles.
@@ -1161,9 +1161,9 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
 				migrate_dirty = !mapping ||
 						mapping->a_ops->migrate_folio;
 			}
-			is_inaccessible = mapping && mapping_inaccessible(mapping);
+			is_unmovable = mapping && mapping_unmovable(mapping);
 			folio_unlock(folio);
-			if (!migrate_dirty || is_inaccessible)
+			if (!migrate_dirty || is_unmovable)
 				goto isolate_fail_put;
 		}
 
diff --git a/mm/migrate.c b/mm/migrate.c
index dd15a84b2a52..d4dcd7f142ce 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -1101,7 +1101,7 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,
 
 	if (!mapping)
 		rc = migrate_folio(mapping, dst, src, mode);
-	else if (mapping_inaccessible(mapping))
+	else if (mapping_unmovable(mapping))
 		rc = -EOPNOTSUPP;
 	else if (mapping->a_ops->migrate_folio)
 		/*
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 45cbdf4801ec..169f75f95433 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -593,6 +593,7 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags)
 	inode->i_size = size;
 	mapping_set_gfp_mask(inode->i_mapping, GFP_HIGHUSER);
 	mapping_set_inaccessible(inode->i_mapping);
+	mapping_set_unmovable(inode->i_mapping);
 	/* Unmovable mappings are supposed to be marked unevictable as well. */
 	WARN_ON_ONCE(!mapping_unevictable(inode->i_mapping));
 

-- 
2.43.0

[PATCH v3 3/9] KVM: guest_memfd: implement folio migration for non-confidential VMs

[Shivank Garg]

Implement kvm_gmem_migrate_folio() using filemap_migrate_folio() for
non-confidential VMs but don't enable it yet. guest_memfd mappings
are still marked unmovable, so MM never calls it.

MM invokes ->migrate_folio() callback with folios already locked and
acquires the invalidate lock afterwards. kvm_gmem_punch_hole() takes
the invalidate lock before locking folios during truncation, so
taking invalidate lock in migration callback can cause an ABBA-deadlock
situation. Use filemap_invalidate_trylock_shared() to avoid this.

A subsequent patch adds an opt-in flag to change the mapping to
movable and enable migration. Non-confidential VMs can use
host-side copy via folio_mc_copy(), whereas confidential VMs will
require firmware-assisted copying before they can opt in.

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 virt/kvm/guest_memfd.c | 36 ++++++++++++++++++++++++++++++++++--
 1 file changed, 34 insertions(+), 2 deletions(-)

diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 169f75f95433..583341d593c9 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -488,13 +488,45 @@ static struct file_operations kvm_gmem_fops = {
 	.fallocate	= kvm_gmem_fallocate,
 };
 
+#ifdef CONFIG_MIGRATION
 static int kvm_gmem_migrate_folio(struct address_space *mapping,
 				  struct folio *dst, struct folio *src,
 				  enum migrate_mode mode)
 {
-	WARN_ON_ONCE(1);
-	return -EINVAL;
+	struct inode *inode = mapping->host;
+	pgoff_t start, end;
+	int ret;
+
+	/*
+	 * Migration invokes ->migrate_folio() while holding the folio lock.
+	 * Use a non-blocking trylock to avoid inverting the lock order with
+	 * truncation, which takes the invalidate lock before locking the
+	 * folios.
+	 */
+	if (!filemap_invalidate_trylock_shared(mapping))
+		return -EAGAIN;
+
+	start = src->index;
+	end = start + folio_nr_pages(src);
+
+	kvm_gmem_invalidate_start(inode, start, end);
+
+	/*
+	 * For non-confidential guests the folio is host-readable, so
+	 * filemap_migrate_folio() can copy the contents itself via
+	 * folio_mc_copy().
+	 * For confidential guests, this would need firmware assistance.
+	 */
+	ret = filemap_migrate_folio(mapping, dst, src, mode);
+
+	kvm_gmem_invalidate_end(inode, start, end);
+
+	filemap_invalidate_unlock_shared(mapping);
+	return ret;
 }
+#else
+#define kvm_gmem_migrate_folio NULL
+#endif
 
 static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *folio)
 {

-- 
2.43.0

Re: [PATCH v3 3/9] KVM: guest_memfd: implement folio migration for non-confidential VMs

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] Pre-existing ABBA deadlock between folio lock and filemap_invalidate_lock in kvm_gmem_error_folio
--

--- Patch [3]: [PATCH v3 3/9] KVM: guest_memfd: implement folio migration for non-confidential VMs ---
commit 7a6778b9d1b7f5633978063ee1c7ee906f8f31c4
Author: Shivank Garg <shivankg@amd.com>

KVM: guest_memfd: implement folio migration for non-confidential VMs

This commit implements the folio migration callback for guest_memfd using
filemap_migrate_folio(). It explicitly uses a trylock for the invalidate lock
to avoid an ABBA deadlock with the truncation path.

> diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
> index 169f75f954337..583341d593c93 100644
> --- a/virt/kvm/guest_memfd.c
> +++ b/virt/kvm/guest_memfd.c
[ ... ]
> @@ -488,13 +488,45 @@ static struct file_operations kvm_gmem_fops = {
[ ... ]
> +	/*
> +	 * Migration invokes ->migrate_folio() while holding the folio lock.
> +	 * Use a non-blocking trylock to avoid inverting the lock order with
> +	 * truncation, which takes the invalidate lock before locking the
> +	 * folios.
> +	 */
> +	if (!filemap_invalidate_trylock_shared(mapping))
> +		return -EAGAIN;
[ ... ]
>  static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *folio)
>  {
>  	pgoff_t start, end;
>  
>  	filemap_invalidate_lock_shared(mapping);

[Severity: High]
This is a pre-existing issue, but does kvm_gmem_error_folio() suffer from the
same ABBA deadlock that you correctly avoided in the new migration path?

The truncation path in kvm_gmem_punch_hole() acquires the invalidate_lock for 
write, and then acquires folio locks during truncate_inode_pages_range().

Concurrently, if a hardware memory error occurs, memory_failure() acquires the
folio lock before calling the error_remove_folio callback (which is routed to
kvm_gmem_error_folio()). This callback then unconditionally calls 
filemap_invalidate_lock_shared(), resulting in an inversion:

Thread 1 (Truncation):
  kvm_gmem_punch_hole()
    filemap_invalidate_lock(inode->i_mapping);
    truncate_inode_pages_range(...)
      folio_lock(folio);

Thread 2 (Memory Failure):
  memory_failure()
    folio_lock(folio);
    kvm_gmem_error_folio(...)
      filemap_invalidate_lock_shared(mapping);

Should this error path also use filemap_invalidate_trylock_shared() or another
mechanism to avoid the deadlock?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260805-shivank-gmem-migrate-v3-0-00d8bdec4e1d@amd.com?part=3

[PATCH v3 4/9] KVM: guest_memfd: add GUEST_MEMFD_FLAG_MIGRATABLE

[Shivank Garg]

guest_memfd folios are currently always marked unmovable, which
prevents the MM from invoking ->migrate_folio callback in scenarios
like compaction, offlining, etc. This is unavoidable for confidential VMs
(SEV-SNP, TDX), since memory is encrypted and copying it needs firmware
assistance. However, for non-confidential VMs (like firecracker), we
can still migrate the folios.

Add kvm_arch_supports_gmem_migration() so an architecture can control
whether migration is supported. Add GUEST_MEMFD_FLAG_MIGRATABLE so
userspace can opt-in to folio migration.

For opted-in guest_memfds, use GFP_HIGHUSER_MOVABLE so their folios can
be allocated from ZONE_MOVABLE/CMA and grouped by mobility, otherwise
keep the existing unmovable behavior.

Remove redundant WARN_ON_ONCE for checking unevictable mapping.

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 Documentation/virt/kvm/api.rst |  3 +++
 arch/x86/kvm/x86.c             |  9 +++++++++
 include/linux/kvm_host.h       |  4 ++++
 include/uapi/linux/kvm.h       |  1 +
 tools/include/uapi/linux/kvm.h |  1 +
 virt/kvm/guest_memfd.c         | 18 ++++++++++++++----
 6 files changed, 32 insertions(+), 4 deletions(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index e3003a241d5b..976921527ef9 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -6461,6 +6461,9 @@ specified via KVM_CREATE_GUEST_MEMFD.  Currently defined flags:
                                without INIT_SHARED will be marked private).
                                Shared memory can be faulted into host userspace
                                page tables. Private memory cannot.
+ GUEST_MEMFD_FLAG_MIGRATABLE   Allow MM to migrate guest_memfd folios.
+                               Availability is architecture sepecific. Without
+                               this flag, folios remain unmovable.
   ============================ ================================================
 
 When the KVM MMU performs a PFN lookup to service a guest fault and the backing
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 47cb9eba113b..7087b0b8f973 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -14130,6 +14130,15 @@ bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm)
 	return !kvm_arch_has_private_mem(kvm);
 }
 
+/*
+ * Migration of guest_memfd with private memory is not supported yet
+ * as this may require architecture-specific handling.
+ */
+bool kvm_arch_supports_gmem_migration(struct kvm *kvm)
+{
+	return !kvm_arch_has_private_mem(kvm);
+}
+
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
 int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_order)
 {
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index ab8cfaec82d3..7b2f15b21970 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -731,6 +731,7 @@ static inline bool kvm_arch_has_private_mem(struct kvm *kvm)
 
 #ifdef CONFIG_KVM_GUEST_MEMFD
 bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm);
+bool kvm_arch_supports_gmem_migration(struct kvm *kvm);
 
 static inline u64 kvm_gmem_get_supported_flags(struct kvm *kvm)
 {
@@ -738,6 +739,9 @@ static inline u64 kvm_gmem_get_supported_flags(struct kvm *kvm)
 
 	if (!kvm || kvm_arch_supports_gmem_init_shared(kvm))
 		flags |= GUEST_MEMFD_FLAG_INIT_SHARED;
+	if (IS_ENABLED(CONFIG_MIGRATION) &&
+	    (!kvm || kvm_arch_supports_gmem_migration(kvm)))
+		flags |= GUEST_MEMFD_FLAG_MIGRATABLE;
 
 	return flags;
 }
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 419011097fa8..e1615a62ed78 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -1654,6 +1654,7 @@ struct kvm_memory_attributes {
 #define KVM_CREATE_GUEST_MEMFD	_IOWR(KVMIO,  0xd4, struct kvm_create_guest_memfd)
 #define GUEST_MEMFD_FLAG_MMAP		(1ULL << 0)
 #define GUEST_MEMFD_FLAG_INIT_SHARED	(1ULL << 1)
+#define GUEST_MEMFD_FLAG_MIGRATABLE	(1ULL << 2)
 
 struct kvm_create_guest_memfd {
 	__u64 size;
diff --git a/tools/include/uapi/linux/kvm.h b/tools/include/uapi/linux/kvm.h
index 419011097fa8..e1615a62ed78 100644
--- a/tools/include/uapi/linux/kvm.h
+++ b/tools/include/uapi/linux/kvm.h
@@ -1654,6 +1654,7 @@ struct kvm_memory_attributes {
 #define KVM_CREATE_GUEST_MEMFD	_IOWR(KVMIO,  0xd4, struct kvm_create_guest_memfd)
 #define GUEST_MEMFD_FLAG_MMAP		(1ULL << 0)
 #define GUEST_MEMFD_FLAG_INIT_SHARED	(1ULL << 1)
+#define GUEST_MEMFD_FLAG_MIGRATABLE	(1ULL << 2)
 
 struct kvm_create_guest_memfd {
 	__u64 size;
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 583341d593c9..a2b7c2f1c49f 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -589,6 +589,11 @@ bool __weak kvm_arch_supports_gmem_init_shared(struct kvm *kvm)
 	return true;
 }
 
+bool __weak kvm_arch_supports_gmem_migration(struct kvm *kvm)
+{
+	return false;
+}
+
 static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags)
 {
 	static const char *name = "[kvm-gmem]";
@@ -623,11 +628,16 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags)
 	inode->i_mapping->a_ops = &kvm_gmem_aops;
 	inode->i_mode |= S_IFREG;
 	inode->i_size = size;
-	mapping_set_gfp_mask(inode->i_mapping, GFP_HIGHUSER);
 	mapping_set_inaccessible(inode->i_mapping);
-	mapping_set_unmovable(inode->i_mapping);
-	/* Unmovable mappings are supposed to be marked unevictable as well. */
-	WARN_ON_ONCE(!mapping_unevictable(inode->i_mapping));
+	/* guest_memfd mappings should be marked unevictable. */
+	mapping_set_unevictable(inode->i_mapping);
+
+	if (flags & GUEST_MEMFD_FLAG_MIGRATABLE) {
+		mapping_set_gfp_mask(inode->i_mapping, GFP_HIGHUSER_MOVABLE);
+	} else {
+		mapping_set_gfp_mask(inode->i_mapping, GFP_HIGHUSER);
+		mapping_set_unmovable(inode->i_mapping);
+	}
 
 	GMEM_I(inode)->flags = flags;
 

-- 
2.43.0

Re: [PATCH v3 4/9] KVM: guest_memfd: add GUEST_MEMFD_FLAG_MIGRATABLE

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] Global capability check for KVM_CAP_GUEST_MEMFD_FLAGS falsely advertises GUEST_MEMFD_FLAG_MIGRATABLE on unsupported architectures.
- [Low] Typo in API documentation for GUEST_MEMFD_FLAG_MIGRATABLE.
--

--- Patch [4]: [PATCH v3 4/9] KVM: guest_memfd: add GUEST_MEMFD_FLAG_MIGRATABLE ---
commit f12765f58cf3ad7c38abcf69c609570a66578e93
Author: Shivank Garg <shivankg@amd.com>

KVM: guest_memfd: add GUEST_MEMFD_FLAG_MIGRATABLE
    
This patch adds architecture support checks and a new flag,
GUEST_MEMFD_FLAG_MIGRATABLE, to allow the MM to migrate guest_memfd folios.
It updates the mapping GFP mask and removes a redundant WARN_ON_ONCE.

> diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
> index e3003a241d5b0..976921527ef9a 100644
> --- a/Documentation/virt/kvm/api.rst
> +++ b/Documentation/virt/kvm/api.rst
> @@ -6461,6 +6461,9 @@ specified via KVM_CREATE_GUEST_MEMFD.  Currently defined flags:
>                                 without INIT_SHARED will be marked private).
>                                 Shared memory can be faulted into host userspace
>                                 page tables. Private memory cannot.
> + GUEST_MEMFD_FLAG_MIGRATABLE   Allow MM to migrate guest_memfd folios.
> +                               Availability is architecture sepecific. Without

[Severity: Low]
This isn't a bug, but there is a typo here (sepecific instead of specific).

> +                               this flag, folios remain unmovable.
>    ============================ ================================================

[ ... ]

> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index 47cb9eba113b1..7087b0b8f973f 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -14130,6 +14130,15 @@ bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm)
>  	return !kvm_arch_has_private_mem(kvm);
>  }
>  
> +/*
> + * Migration of guest_memfd with private memory is not supported yet
> + * as this may require architecture-specific handling.
> + */
> +bool kvm_arch_supports_gmem_migration(struct kvm *kvm)
> +{
> +	return !kvm_arch_has_private_mem(kvm);
> +}

[ ... ]

> diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
> index ab8cfaec82d31..7b2f15b219705 100644
> --- a/include/linux/kvm_host.h
> +++ b/include/linux/kvm_host.h
> @@ -731,6 +731,7 @@ static inline bool kvm_arch_has_private_mem(struct kvm *kvm)
>  
>  #ifdef CONFIG_KVM_GUEST_MEMFD
>  bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm);
> +bool kvm_arch_supports_gmem_migration(struct kvm *kvm);
>  
>  static inline u64 kvm_gmem_get_supported_flags(struct kvm *kvm)
>  {
> @@ -738,6 +739,9 @@ static inline u64 kvm_gmem_get_supported_flags(struct kvm *kvm)
>  
>  	if (!kvm || kvm_arch_supports_gmem_init_shared(kvm))
>  		flags |= GUEST_MEMFD_FLAG_INIT_SHARED;
> +	if (IS_ENABLED(CONFIG_MIGRATION) &&
> +	    (!kvm || kvm_arch_supports_gmem_migration(kvm)))
> +		flags |= GUEST_MEMFD_FLAG_MIGRATABLE;

[Severity: Medium]
When userspace queries KVM_CAP_GUEST_MEMFD_FLAGS on the global /dev/kvm
file descriptor, kvm is NULL. Does the !kvm check here cause the API to
unconditionally advertise GUEST_MEMFD_FLAG_MIGRATABLE for all architectures
where CONFIG_MIGRATION is enabled?

If so, architectures that do not support migration (relying on the weak
default returning false) would advertise support globally, but then return
-EINVAL when userspace actually attempts to create a guest_memfd with this
flag.

If we drop the !kvm bypass to fix the capability advertisement, would the
x86 implementation of kvm_arch_supports_gmem_migration() shown above need an
explicit check for a NULL kvm pointer before passing it to
kvm_arch_has_private_mem()?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260805-shivank-gmem-migrate-v3-0-00d8bdec4e1d@amd.com?part=4

[PATCH v3 5/9] KVM: selftests: fix maxnode arguments in xapic_ipi_test

[Shivank Garg]

migrate_pages() syscall expect maxnode to be one greater than the
number of bits in the nodemask. do_migrations() passes the size of
nodemask in bytes to migrate_pages(). This sets the maxnode to 8,
so kernel only checks node IDs 0-6 even though the nodemask covers
node IDs 0-63.

Pass the nodemask size in bits plus one.

Fixes: 678e90a349a4 ("KVM: selftests: Test IPI to halted vCPU in xAPIC while backing page moves")
Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 tools/testing/selftests/kvm/x86/xapic_ipi_test.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/testing/selftests/kvm/x86/xapic_ipi_test.c b/tools/testing/selftests/kvm/x86/xapic_ipi_test.c
index 39ce9a9369f5..761e47e4cae2 100644
--- a/tools/testing/selftests/kvm/x86/xapic_ipi_test.c
+++ b/tools/testing/selftests/kvm/x86/xapic_ipi_test.c
@@ -310,7 +310,7 @@ void do_migrations(struct test_data_page *data, int run_secs, int delay_usecs,
 		 * KVM_CREATE_VCPU ioctl. If that assumption ever changes this
 		 * test may break or give a false positive signal.
 		 */
-		pages_not_moved = migrate_pages(0, sizeof(nodemasks[from]),
+		pages_not_moved = migrate_pages(0, sizeof(nodemasks[from]) * 8 + 1,
 						&nodemasks[from],
 						&nodemasks[to]);
 		if (pages_not_moved < 0)

-- 
2.43.0

[PATCH v3 6/9] KVM: selftests: use BITS_PER_TYPE() for NUMA masks

[Shivank Garg]

Replace the open-coded sizeof() * 8 calculations in do_migrations() with
BITS_PER_TYPE().

No functional change intended.

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 tools/testing/selftests/kvm/x86/xapic_ipi_test.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/tools/testing/selftests/kvm/x86/xapic_ipi_test.c b/tools/testing/selftests/kvm/x86/xapic_ipi_test.c
index 761e47e4cae2..769d8d95ab2c 100644
--- a/tools/testing/selftests/kvm/x86/xapic_ipi_test.c
+++ b/tools/testing/selftests/kvm/x86/xapic_ipi_test.c
@@ -252,7 +252,7 @@ void do_migrations(struct test_data_page *data, int run_secs, int delay_usecs,
 {
 	long pages_not_moved;
 	unsigned long nodemask = 0;
-	unsigned long nodemasks[sizeof(nodemask) * 8];
+	unsigned long nodemasks[BITS_PER_TYPE(nodemask)];
 	int nodes = 0;
 	time_t start_time, last_update, now;
 	time_t interval_secs = 1;
@@ -267,18 +267,18 @@ void do_migrations(struct test_data_page *data, int run_secs, int delay_usecs,
 		delay_usecs);
 
 	/* Get set of first 64 numa nodes available */
-	kvm_get_mempolicy(NULL, &nodemask, sizeof(nodemask) * 8,
+	kvm_get_mempolicy(NULL, &nodemask, BITS_PER_TYPE(nodemask),
 			  0, MPOL_F_MEMS_ALLOWED);
 
 	fprintf(stderr, "Numa nodes found amongst first %lu possible nodes "
 		"(each 1-bit indicates node is present): %#lx\n",
-		sizeof(nodemask) * 8, nodemask);
+		BITS_PER_TYPE(nodemask), nodemask);
 
 	/* Init array of masks containing a single-bit in each, one for each
 	 * available node. migrate_pages called below requires specifying nodes
 	 * as bit masks.
 	 */
-	for (i = 0, bit = 1; i < sizeof(nodemask) * 8; i++, bit <<= 1) {
+	for (i = 0, bit = 1; i < BITS_PER_TYPE(nodemask); i++, bit <<= 1) {
 		if (nodemask & bit) {
 			nodemasks[nodes] = nodemask & bit;
 			nodes++;
@@ -310,7 +310,7 @@ void do_migrations(struct test_data_page *data, int run_secs, int delay_usecs,
 		 * KVM_CREATE_VCPU ioctl. If that assumption ever changes this
 		 * test may break or give a false positive signal.
 		 */
-		pages_not_moved = migrate_pages(0, sizeof(nodemasks[from]) * 8 + 1,
+		pages_not_moved = migrate_pages(0, BITS_PER_TYPE(nodemasks[from]) + 1,
 						&nodemasks[from],
 						&nodemasks[to]);
 		if (pages_not_moved < 0)

-- 
2.43.0

[PATCH v3 7/9] KVM: selftests: add get_numa_mem_nodes()

[Shivank Garg]

The xAPIC IPI test uses MPOL_F_MEMS_ALLOWED to get the memory nodes
available to the current process. Move the query to numaif.h as
get_numa_mem_nodes() so other KVM selftests can use it.

Call get_mempolicy() directly instead of using the assert-on-failure
kvm_get_mempolicy() wrapper. The xAPIC test already asserts its two
node requirement.

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 tools/testing/selftests/kvm/include/numaif.h     | 12 ++++++++++++
 tools/testing/selftests/kvm/x86/xapic_ipi_test.c |  6 ++----
 2 files changed, 14 insertions(+), 4 deletions(-)

diff --git a/tools/testing/selftests/kvm/include/numaif.h b/tools/testing/selftests/kvm/include/numaif.h
index 29572a6d789c..55124e2330ab 100644
--- a/tools/testing/selftests/kvm/include/numaif.h
+++ b/tools/testing/selftests/kvm/include/numaif.h
@@ -75,6 +75,18 @@ static bool is_numa_available(void)
 		(errno != ENOSYS && errno != EPERM);
 }
 
+static inline unsigned long get_numa_mem_nodes(void)
+{
+	unsigned long nodemask = 0;
+
+	/* Get set of first 64 numa nodes available */
+	if (get_mempolicy(NULL, &nodemask, BITS_PER_TYPE(nodemask), NULL,
+			  MPOL_F_MEMS_ALLOWED))
+		return 0;
+
+	return nodemask;
+}
+
 static inline bool is_multi_numa_node_system(void)
 {
 	return is_numa_available() && get_max_numa_node() >= 1;
diff --git a/tools/testing/selftests/kvm/x86/xapic_ipi_test.c b/tools/testing/selftests/kvm/x86/xapic_ipi_test.c
index 769d8d95ab2c..66dcf36398aa 100644
--- a/tools/testing/selftests/kvm/x86/xapic_ipi_test.c
+++ b/tools/testing/selftests/kvm/x86/xapic_ipi_test.c
@@ -251,7 +251,7 @@ void do_migrations(struct test_data_page *data, int run_secs, int delay_usecs,
 		   u64 *pipis_rcvd)
 {
 	long pages_not_moved;
-	unsigned long nodemask = 0;
+	unsigned long nodemask;
 	unsigned long nodemasks[BITS_PER_TYPE(nodemask)];
 	int nodes = 0;
 	time_t start_time, last_update, now;
@@ -266,9 +266,7 @@ void do_migrations(struct test_data_page *data, int run_secs, int delay_usecs,
 	fprintf(stderr, "Calling migrate_pages every %d microseconds\n",
 		delay_usecs);
 
-	/* Get set of first 64 numa nodes available */
-	kvm_get_mempolicy(NULL, &nodemask, BITS_PER_TYPE(nodemask),
-			  0, MPOL_F_MEMS_ALLOWED);
+	nodemask = get_numa_mem_nodes();
 
 	fprintf(stderr, "Numa nodes found amongst first %lu possible nodes "
 		"(each 1-bit indicates node is present): %#lx\n",

-- 
2.43.0

[PATCH v3 8/9] KVM: selftests: use allowed NUMA nodes in guest_memfd_test

[Shivank Garg]

guest_memfd_test assumes that nodes 0 and 1 exist and have memory.
is_multi_numa_node_system() only checks that the maximum node ID is
nonzero, which is not enough for sparse or memoryless nodes.

Select the required nodes from MPOL_F_MEMS_ALLOWED instead. Use the full
nodemask width plus one to mbind(), and let test_mbind() run when only
one memory node is available.

The sysfs helpers for finding maxnode are no longer needed.

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 tools/testing/selftests/kvm/guest_memfd_test.c | 86 +++++++++++++++++---------
 tools/testing/selftests/kvm/include/numaif.h   | 52 ----------------
 2 files changed, 58 insertions(+), 80 deletions(-)

diff --git a/tools/testing/selftests/kvm/guest_memfd_test.c b/tools/testing/selftests/kvm/guest_memfd_test.c
index 2233d871a38f..aee80dda6229 100644
--- a/tools/testing/selftests/kvm/guest_memfd_test.c
+++ b/tools/testing/selftests/kvm/guest_memfd_test.c
@@ -76,33 +76,53 @@ static void test_mmap_supported(int fd, size_t total_size)
 	kvm_munmap(mem, total_size);
 }
 
+/*
+ * Fill @nids with the first @nr_nids nodes in the allowed mask.
+ * Return false if the mask contains fewer than @nr_nids nodes.
+ */
+static bool get_numa_node_ids(int *nids, int nr_nids)
+{
+	unsigned long nodemask = get_numa_mem_nodes();
+	unsigned long nid;
+	int nr_found = 0;
+
+	for_each_set_bit(nid, &nodemask, BITS_PER_TYPE(nodemask)) {
+		nids[nr_found++] = nid;
+		if (nr_found == nr_nids)
+			return true;
+	}
+
+	return false;
+}
+
 static void test_mbind(int fd, size_t total_size)
 {
-	const unsigned long nodemask_0 = 1; /* nid: 0 */
-	unsigned long nodemask = 0;
-	unsigned long maxnode = BITS_PER_TYPE(nodemask);
+	unsigned long nodemask, bind_nodemask;
+	unsigned long maxnode = BITS_PER_TYPE(nodemask) + 1;
 	int policy;
 	char *mem;
+	int nid;
 	int ret;
 
-	if (!is_multi_numa_node_system())
+	if (!get_numa_node_ids(&nid, 1))
 		return;
 
+	bind_nodemask = 1UL << nid;
 	mem = kvm_mmap(total_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd);
 
 	/* Test MPOL_INTERLEAVE policy */
-	kvm_mbind(mem, page_size * 2, MPOL_INTERLEAVE, &nodemask_0, maxnode, 0);
+	kvm_mbind(mem, page_size * 2, MPOL_INTERLEAVE, &bind_nodemask, maxnode, 0);
 	kvm_get_mempolicy(&policy, &nodemask, maxnode, mem, MPOL_F_ADDR);
-	TEST_ASSERT(policy == MPOL_INTERLEAVE && nodemask == nodemask_0,
+	TEST_ASSERT(policy == MPOL_INTERLEAVE && nodemask == bind_nodemask,
 		    "Wanted MPOL_INTERLEAVE (%u) and nodemask 0x%lx, got %u and 0x%lx",
-		    MPOL_INTERLEAVE, nodemask_0, policy, nodemask);
+		    MPOL_INTERLEAVE, bind_nodemask, policy, nodemask);
 
 	/* Test basic MPOL_BIND policy */
-	kvm_mbind(mem + page_size * 2, page_size * 2, MPOL_BIND, &nodemask_0, maxnode, 0);
+	kvm_mbind(mem + page_size * 2, page_size * 2, MPOL_BIND, &bind_nodemask, maxnode, 0);
 	kvm_get_mempolicy(&policy, &nodemask, maxnode, mem + page_size * 2, MPOL_F_ADDR);
-	TEST_ASSERT(policy == MPOL_BIND && nodemask == nodemask_0,
+	TEST_ASSERT(policy == MPOL_BIND && nodemask == bind_nodemask,
 		    "Wanted MPOL_BIND (%u) and nodemask 0x%lx, got %u and 0x%lx",
-		    MPOL_BIND, nodemask_0, policy, nodemask);
+		    MPOL_BIND, bind_nodemask, policy, nodemask);
 
 	/* Test MPOL_DEFAULT policy */
 	kvm_mbind(mem, total_size, MPOL_DEFAULT, NULL, 0, 0);
@@ -112,7 +132,7 @@ static void test_mbind(int fd, size_t total_size)
 		    MPOL_DEFAULT, policy, nodemask);
 
 	/* Test with invalid policy */
-	ret = mbind(mem, page_size, 999, &nodemask_0, maxnode, 0);
+	ret = mbind(mem, page_size, 999, &bind_nodemask, maxnode, 0);
 	TEST_ASSERT(ret == -1 && errno == EINVAL,
 		    "mbind with invalid policy should fail with EINVAL");
 
@@ -121,17 +141,19 @@ static void test_mbind(int fd, size_t total_size)
 
 static void test_numa_allocation(int fd, size_t total_size)
 {
-	unsigned long node0_mask = 1;  /* Node 0 */
-	unsigned long node1_mask = 2;  /* Node 1 */
-	unsigned long maxnode = 8;
+	unsigned long bind_nodemasks[2];
+	unsigned long maxnode = BITS_PER_TYPE(bind_nodemasks[0]) + 1;
 	void *pages[4];
+	int nids[2];
 	int status[4];
 	char *mem;
 	int i;
 
-	if (!is_multi_numa_node_system())
+	if (!get_numa_node_ids(nids, ARRAY_SIZE(nids)))
 		return;
 
+	bind_nodemasks[0] = 1UL << nids[0];
+	bind_nodemasks[1] = 1UL << nids[1];
 	mem = kvm_mmap(total_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd);
 
 	for (i = 0; i < 4; i++)
@@ -139,34 +161,42 @@ static void test_numa_allocation(int fd, size_t total_size)
 
 	/* Set NUMA policy after allocation */
 	memset(mem, 0xaa, page_size);
-	kvm_mbind(pages[0], page_size, MPOL_BIND, &node0_mask, maxnode, 0);
+	kvm_mbind(pages[0], page_size, MPOL_BIND, &bind_nodemasks[0], maxnode, 0);
 	kvm_fallocate(fd, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, 0, page_size);
 
 	/* Set NUMA policy before allocation */
-	kvm_mbind(pages[0], page_size * 2, MPOL_BIND, &node1_mask, maxnode, 0);
-	kvm_mbind(pages[2], page_size * 2, MPOL_BIND, &node0_mask, maxnode, 0);
+	kvm_mbind(pages[0], page_size * 2, MPOL_BIND, &bind_nodemasks[1], maxnode, 0);
+	kvm_mbind(pages[2], page_size * 2, MPOL_BIND, &bind_nodemasks[0], maxnode, 0);
 	memset(mem, 0xaa, total_size);
 
 	/* Validate if pages are allocated on specified NUMA nodes */
 	kvm_move_pages(0, 4, pages, NULL, status, 0);
-	TEST_ASSERT(status[0] == 1, "Expected page 0 on node 1, got it on node %d", status[0]);
-	TEST_ASSERT(status[1] == 1, "Expected page 1 on node 1, got it on node %d", status[1]);
-	TEST_ASSERT(status[2] == 0, "Expected page 2 on node 0, got it on node %d", status[2]);
-	TEST_ASSERT(status[3] == 0, "Expected page 3 on node 0, got it on node %d", status[3]);
+	TEST_ASSERT(status[0] == nids[1],
+		    "Expected page 0 on node %d, got it on node %d", nids[1], status[0]);
+	TEST_ASSERT(status[1] == nids[1],
+		    "Expected page 1 on node %d, got it on node %d", nids[1], status[1]);
+	TEST_ASSERT(status[2] == nids[0],
+		    "Expected page 2 on node %d, got it on node %d", nids[0], status[2]);
+	TEST_ASSERT(status[3] == nids[0],
+		    "Expected page 3 on node %d, got it on node %d", nids[0], status[3]);
 
 	/* Punch hole for all pages */
 	kvm_fallocate(fd, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, 0, total_size);
 
 	/* Change NUMA policy nodes and reallocate */
-	kvm_mbind(pages[0], page_size * 2, MPOL_BIND, &node0_mask, maxnode, 0);
-	kvm_mbind(pages[2], page_size * 2, MPOL_BIND, &node1_mask, maxnode, 0);
+	kvm_mbind(pages[0], page_size * 2, MPOL_BIND, &bind_nodemasks[0], maxnode, 0);
+	kvm_mbind(pages[2], page_size * 2, MPOL_BIND, &bind_nodemasks[1], maxnode, 0);
 	memset(mem, 0xaa, total_size);
 
 	kvm_move_pages(0, 4, pages, NULL, status, 0);
-	TEST_ASSERT(status[0] == 0, "Expected page 0 on node 0, got it on node %d", status[0]);
-	TEST_ASSERT(status[1] == 0, "Expected page 1 on node 0, got it on node %d", status[1]);
-	TEST_ASSERT(status[2] == 1, "Expected page 2 on node 1, got it on node %d", status[2]);
-	TEST_ASSERT(status[3] == 1, "Expected page 3 on node 1, got it on node %d", status[3]);
+	TEST_ASSERT(status[0] == nids[0],
+		    "Expected page 0 on node %d, got it on node %d", nids[0], status[0]);
+	TEST_ASSERT(status[1] == nids[0],
+		    "Expected page 1 on node %d, got it on node %d", nids[0], status[1]);
+	TEST_ASSERT(status[2] == nids[1],
+		    "Expected page 2 on node %d, got it on node %d", nids[1], status[2]);
+	TEST_ASSERT(status[3] == nids[1],
+		    "Expected page 3 on node %d, got it on node %d", nids[1], status[3]);
 
 	kvm_munmap(mem, total_size);
 }
diff --git a/tools/testing/selftests/kvm/include/numaif.h b/tools/testing/selftests/kvm/include/numaif.h
index 55124e2330ab..4bbbf314e9d7 100644
--- a/tools/testing/selftests/kvm/include/numaif.h
+++ b/tools/testing/selftests/kvm/include/numaif.h
@@ -4,8 +4,6 @@
 #ifndef SELFTEST_KVM_NUMAIF_H
 #define SELFTEST_KVM_NUMAIF_H
 
-#include <dirent.h>
-
 #include <linux/mempolicy.h>
 
 #include "kvm_syscalls.h"
@@ -30,51 +28,6 @@ KVM_SYSCALL_DEFINE(mbind, 6, void *, addr, unsigned long, size, int, mode,
 		   const unsigned long *, nodemask, unsigned long, maxnode,
 		   unsigned int, flags);
 
-static inline int get_max_numa_node(void)
-{
-	struct dirent *de;
-	int max_node = 0;
-	DIR *d;
-
-	/*
-	 * Assume there's a single node if the kernel doesn't support NUMA,
-	 * or if no nodes are found.
-	 */
-	d = opendir("/sys/devices/system/node");
-	if (!d)
-		return 0;
-
-	while ((de = readdir(d)) != NULL) {
-		int node_id;
-		char *endptr;
-
-		if (strncmp(de->d_name, "node", 4) != 0)
-			continue;
-
-		node_id = strtol(de->d_name + 4, &endptr, 10);
-		if (*endptr != '\0')
-			continue;
-
-		if (node_id > max_node)
-			max_node = node_id;
-	}
-	closedir(d);
-
-	return max_node;
-}
-
-static bool is_numa_available(void)
-{
-	/*
-	 * Probe for NUMA by doing a dummy get_mempolicy().  If the syscall
-	 * fails with ENOSYS, then the kernel was built without NUMA support.
-	 * if the syscall fails with EPERM, then the process/user lacks the
-	 * necessary capabilities (CAP_SYS_NICE).
-	 */
-	return !get_mempolicy(NULL, NULL, 0, NULL, 0) ||
-		(errno != ENOSYS && errno != EPERM);
-}
-
 static inline unsigned long get_numa_mem_nodes(void)
 {
 	unsigned long nodemask = 0;
@@ -87,9 +40,4 @@ static inline unsigned long get_numa_mem_nodes(void)
 	return nodemask;
 }
 
-static inline bool is_multi_numa_node_system(void)
-{
-	return is_numa_available() && get_max_numa_node() >= 1;
-}
-
 #endif /* SELFTEST_KVM_NUMAIF_H */

-- 
2.43.0

[PATCH v3 9/9] KVM: selftests: exercise guest_memfd folio migration

[Shivank Garg]

Add a move_pages() test for guest_memfd. Allocate the folios on one NUMA
node, move them to another node and back, and verify their location and
contents after each move.

Migration is opt-in so verify that migration fails without
GUEST_MEMFD_FLAG_MIGRATABLE and succeeds with the flag set. Also run
the existing guest_memfd tests with the flag set.

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 tools/testing/selftests/kvm/guest_memfd_test.c | 107 +++++++++++++++++++++++++
 1 file changed, 107 insertions(+)

diff --git a/tools/testing/selftests/kvm/guest_memfd_test.c b/tools/testing/selftests/kvm/guest_memfd_test.c
index aee80dda6229..6e1c551fec80 100644
--- a/tools/testing/selftests/kvm/guest_memfd_test.c
+++ b/tools/testing/selftests/kvm/guest_memfd_test.c
@@ -201,6 +201,102 @@ static void test_numa_allocation(int fd, size_t total_size)
 	kvm_munmap(mem, total_size);
 }
 
+static void verify_pages_on_node(char **pages, int page_count, int *status,
+				 int nid, const char *when)
+{
+	int i;
+
+	kvm_move_pages(0, page_count, pages, NULL, status, 0);
+
+	for (i = 0; i < page_count; i++) {
+		char expected = i;
+		size_t off;
+
+		TEST_ASSERT(status[i] == nid,
+			    "Expected page %d on node %d %s, got it on node %d",
+			    i, nid, when, status[i]);
+
+		for (off = 0; off < page_size; off++)
+			TEST_ASSERT(pages[i][off] == expected,
+				    "Page %d corrupted at offset %zu %s",
+				    i, off, when);
+	}
+}
+
+static void move_pages_to_node(char **pages, int page_count, int *nodes,
+			       int *status, int nid)
+{
+	int i;
+
+	for (i = 0; i < page_count; i++)
+		nodes[i] = nid;
+
+	kvm_move_pages(0, page_count, pages, nodes, status, MPOL_MF_MOVE);
+}
+
+static void __test_migrate_folio(int fd, size_t total_size, bool migratable)
+{
+	int page_count = total_size / page_size;
+	unsigned long nodemask;
+	unsigned long maxnode = BITS_PER_TYPE(nodemask) + 1;
+	int *nodes, *status;
+	char **pages;
+	char *mem;
+	int nids[2];
+	int i;
+
+	if (!get_numa_node_ids(nids, ARRAY_SIZE(nids)))
+		return;
+
+	mem = kvm_mmap(total_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd);
+
+	pages = calloc(page_count, sizeof(*pages));
+	nodes = calloc(page_count, sizeof(*nodes));
+	status = calloc(page_count, sizeof(*status));
+	TEST_ASSERT(pages && nodes && status, "Failed to allocate page arrays");
+
+	/* Allocate all folios on the source node and fill in a known pattern. */
+	nodemask = 1UL << nids[0];
+	kvm_mbind(mem, total_size, MPOL_BIND, &nodemask, maxnode, 0);
+	for (i = 0; i < page_count; i++) {
+		pages[i] = mem + i * page_size;
+		memset(pages[i], i, page_size);
+	}
+	verify_pages_on_node(pages, page_count, status, nids[0], "on allocation");
+
+	if (migratable) {
+		move_pages_to_node(pages, page_count, nodes, status, nids[1]);
+		verify_pages_on_node(pages, page_count, status, nids[1],
+				     "after migration");
+
+		move_pages_to_node(pages, page_count, nodes, status, nids[0]);
+		verify_pages_on_node(pages, page_count, status, nids[0],
+				     "after round-trip");
+	} else {
+		for (i = 0; i < page_count; i++)
+			nodes[i] = nids[1];
+		TEST_ASSERT_EQ(move_pages(0, page_count, pages, nodes, status,
+					  MPOL_MF_MOVE), page_count);
+		verify_pages_on_node(pages, page_count, status, nids[0],
+				     "after rejected migration");
+	}
+
+	free(status);
+	free(nodes);
+	free(pages);
+	kvm_munmap(mem, total_size);
+}
+
+static void test_migrate_folio_supported(int fd, size_t total_size)
+{
+	__test_migrate_folio(fd, total_size, true);
+}
+
+static void test_migrate_folio_not_supported(int fd, size_t total_size)
+{
+	__test_migrate_folio(fd, total_size, false);
+}
+
 static void test_collapse(int fd, u64 flags)
 {
 	const size_t pmd_size = get_trans_hugepagesz();
@@ -483,6 +579,11 @@ static void __test_guest_memfd(struct kvm_vm *vm, u64 flags)
 			gmem_test(fault_overflow, vm, flags);
 			gmem_test(numa_allocation, vm, flags);
 			__gmem_test(collapse, vm, flags, pmd_size);
+
+			if (flags & GUEST_MEMFD_FLAG_MIGRATABLE)
+				gmem_test(migrate_folio_supported, vm, flags);
+			else
+				gmem_test(migrate_folio_not_supported, vm, flags);
 		} else {
 			gmem_test(fault_private, vm, flags);
 		}
@@ -501,6 +602,9 @@ static void __test_guest_memfd(struct kvm_vm *vm, u64 flags)
 
 static void test_guest_memfd(unsigned long vm_type)
 {
+	const u64 migratable_flags = GUEST_MEMFD_FLAG_MMAP |
+				     GUEST_MEMFD_FLAG_INIT_SHARED |
+				     GUEST_MEMFD_FLAG_MIGRATABLE;
 	struct kvm_vm *vm = vm_create_barebones_type(vm_type);
 	u64 flags;
 
@@ -517,6 +621,9 @@ static void test_guest_memfd(unsigned long vm_type)
 		__test_guest_memfd(vm, GUEST_MEMFD_FLAG_MMAP |
 				       GUEST_MEMFD_FLAG_INIT_SHARED);
 
+	if ((flags & migratable_flags) == migratable_flags)
+		__test_guest_memfd(vm, migratable_flags);
+
 	kvm_vm_free(vm);
 }
 

-- 
2.43.0