[RFC PATCH 00/11] KVM: Allow alternative providers of guest_memfd backed by PFNMAP memory

[RFC PATCH 00/11] KVM: Allow alternative providers of guest_memfd backed by PFNMAP memory

[David Woodhouse]

For dedicated hosting environments, there are a few reasons to prefer
using external PFNMAP memory for guests, rather than kernel-managed
memory:

 • No struct page overheads

 • Easily managed in 1GiB pages (or larger chunks); no fragmentation/THP

 • Faster kexec/KHO live update (every millisecond spend faffing around
   with memory management is an extra millisecond of steal time taken
   from guests)

Today, there is only one implementation of guest_memfd: the internal
shmem-based page cache in virt/kvm/guest_memfd.c. This series allows
for other code to provide "a guest_memfd". This is an early path-
finding proof of concept tested with AMD SEV-SNP as well as non-CoCo
guests on x86 and arm64. I'm planning to build a memory-based file
system which gives files for both guest_memfd as well as memory-based
storage to allow things like userspace VM and orchestrator stateallows
to be passed over KHO/kexec.

This series lifts guest_memfd's operations into a small ops struct and
converts KVM's own gmem to use it, so it becomes one implementation
among peers rather than the only game in town.  It also adapts the
SEV-SNP fast paths to page-less operation, plumbs iommufd to map a
guest_memfd-backing dma-buf as CPU RAM (not MMIO), and adds the
provider->KVM revocation primitive that makes overcommit-style reclaim
possible.

It's largely AI-built at the moment. My main design concerns are around
that we tell that a given file is "a guest_memfd", and how we plug into
IOMMUFD. Presenting a dma-buf was the less intrusive choice because that
path already has support for taking pages away, but I'm less convinced
that it's the cleaner choice in the long term.

I'll continue to bikeshed it myself, but this is at least functional
enough to show the direction and solicit further opinions...

https://git.infradead.org/?p=users/dwmw2/linux.git;a=shortlog;h=gmem-provider

Connor Williamson (1):
      KVM: SEV: Remove struct page dependency from SNP gmem paths

David Woodhouse (10):
      KVM: selftests: sev_smoke_test: Only run VM types the host offers
      KVM: selftests: sev_init2_tests: Derive SEV availability from KVM
      KVM: guest_memfd: Introduce guest memory ops and route native gmem through them
      iommufd: Look up private-interconnect phys via exporter symbols
      iommufd: Plumb dma-buf memory-type (RAM vs MMIO) through the phys map
      KVM: guest_memfd: Add ops-driven page revocation
      samples/kvm: Add guest_memfd backing sample
      selftests/kvm: gmem_provider KVM-only tests
      selftests/kvm: gmem_provider iommufd tests
      samples/kvm, selftests/kvm: Allow the gmem_provider NVMe DMA test on arm64


 arch/x86/kvm/Makefile                              |   4 +-
 arch/x86/kvm/svm/sev.c                             |  83 ++-
 arch/x86/virt/svm/sev.c                            |  27 +-
 drivers/iommu/iommufd/io_pagetable.h               |   7 +
 drivers/iommu/iommufd/pages.c                      |  49 +-
 include/linux/kvm_host.h                           |  91 +++
 samples/Kconfig                                    |  18 +
 samples/Makefile                                   |   1 +
 samples/kvm/Makefile                               |   2 +
 samples/kvm/gmem_provider.c                        | 651 +++++++++++++++++++++
 samples/kvm/gmem_provider.h                        |  53 ++
 tools/testing/selftests/kvm/Makefile.kvm           |   6 +
 .../selftests/kvm/gmem_provider_nvme_dma_test.c    | 365 ++++++++++++
 .../kvm/x86/gmem_provider_hugepage_test.c          | 130 ++++
 .../selftests/kvm/x86/gmem_provider_iommufd_test.c | 150 +++++
 .../selftests/kvm/x86/gmem_provider_revoke_test.c  | 134 +++++
 .../testing/selftests/kvm/x86/gmem_provider_test.c | 195 ++++++
 .../selftests/kvm/x86/gmem_provider_vfio_test.c    | 134 +++++
 tools/testing/selftests/kvm/x86/sev_init2_tests.c  |  16 +-
 tools/testing/selftests/kvm/x86/sev_smoke_test.c   |   9 +-
 virt/kvm/guest_memfd.c                             | 377 +++++++++---
 virt/kvm/kvm_main.c                                |   6 +-
 virt/kvm/kvm_mm.h                                  |   4 +-
 23 files changed, 2410 insertions(+), 102 deletions(-)

[RFC PATCH 01/11] KVM: selftests: sev_smoke_test: Only run VM types the host offers

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

sev_smoke_test ran the plain SEV subtest unconditionally, gated only on
the X86_FEATURE_SEV CPUID bit, while gating SEV-ES and SNP on the
KVM_CAP_VM_TYPES bits.  CPUID reporting SEV does not mean KVM offers the
SEV VM type: when all SEV ASIDs are assigned to SEV-SNP, KVM_X86_SEV_VM
is unavailable even though X86_FEATURE_SEV is set.  On such a host the
test aborts in KVM_CREATE_VM instead of exercising the available modes.

Gate the SEV subtest on KVM_CAP_VM_TYPES like the others, so the test
runs the VM types the host actually offers.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 tools/testing/selftests/kvm/x86/sev_smoke_test.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/tools/testing/selftests/kvm/x86/sev_smoke_test.c b/tools/testing/selftests/kvm/x86/sev_smoke_test.c
index 6b2cbe2a90b7..bf27b6187afa 100644
--- a/tools/testing/selftests/kvm/x86/sev_smoke_test.c
+++ b/tools/testing/selftests/kvm/x86/sev_smoke_test.c
@@ -247,7 +247,14 @@ int main(int argc, char *argv[])
 {
 	TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_SEV));
 
-	test_sev_smoke(guest_sev_code, KVM_X86_SEV_VM, 0);
+	/*
+	 * Only exercise VM types the host actually offers.  CPUID reporting
+	 * SEV does not guarantee KVM offers the SEV VM type: when all SEV
+	 * ASIDs are assigned to SEV-SNP, KVM_X86_SEV_VM is unavailable even
+	 * though X86_FEATURE_SEV is set.  Gate every type on KVM_CAP_VM_TYPES.
+	 */
+	if (kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SEV_VM))
+		test_sev_smoke(guest_sev_code, KVM_X86_SEV_VM, 0);
 
 	if (kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SEV_ES_VM))
 		test_sev_smoke(guest_sev_es_code, KVM_X86_SEV_ES_VM, SEV_POLICY_ES);

base-commit: 0e35b9b6ec0ffcc5e23cbdec09f5c622ad532b53
-- 
2.54.0

[RFC PATCH 02/11] KVM: selftests: sev_init2_tests: Derive SEV availability from KVM

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

The test asserted that the X86_FEATURE_SEV CPUID bit exactly matches
whether KVM offers KVM_X86_SEV_VM.  That is not an invariant: when all
SEV ASIDs are assigned to SEV-SNP, KVM does not offer the SEV VM type
even though CPUID reports SEV, so the test aborts on an SNP-only host.

Derive SEV availability from KVM_CAP_VM_TYPES (as already done for SEV-ES
and SNP), assert only the one-way implication that a type offered by KVM
is also reported in CPUID, and TEST_REQUIRE() the SEV VM type so the test
skips cleanly when it is unavailable.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 .../testing/selftests/kvm/x86/sev_init2_tests.c  | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/tools/testing/selftests/kvm/x86/sev_init2_tests.c b/tools/testing/selftests/kvm/x86/sev_init2_tests.c
index 8db88c355f16..689390c10f7c 100644
--- a/tools/testing/selftests/kvm/x86/sev_init2_tests.c
+++ b/tools/testing/selftests/kvm/x86/sev_init2_tests.c
@@ -130,12 +130,18 @@ int main(int argc, char *argv[])
 			    KVM_X86_SEV_VMSA_FEATURES,
 			    &supported_vmsa_features);
 
-	have_sev = kvm_cpu_has(X86_FEATURE_SEV);
-	TEST_ASSERT(have_sev == !!(kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SEV_VM)),
-		    "sev: KVM_CAP_VM_TYPES (%x) does not match cpuid (checking %x)",
-		    kvm_check_cap(KVM_CAP_VM_TYPES), 1 << KVM_X86_SEV_VM);
+	/*
+	 * Whether a VM type is available depends on KVM, not just CPUID: e.g.
+	 * when all SEV ASIDs are assigned to SEV-SNP, KVM does not offer the
+	 * SEV VM type even though X86_FEATURE_SEV is set.  Derive availability
+	 * from KVM_CAP_VM_TYPES and only assert the one-way implication that a
+	 * type offered by KVM must also be reported in CPUID.
+	 */
+	have_sev = kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SEV_VM);
+	TEST_ASSERT(!have_sev || kvm_cpu_has(X86_FEATURE_SEV),
+		    "sev: SEV_VM supported without SEV in CPUID");
 
-	TEST_REQUIRE(kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SEV_VM));
+	TEST_REQUIRE(have_sev);
 	have_sev_es = kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SEV_ES_VM);
 
 	TEST_ASSERT(!have_sev_es || kvm_cpu_has(X86_FEATURE_SEV_ES),
-- 
2.54.0

[RFC PATCH 03/11] KVM: SEV: Remove struct page dependency from SNP gmem paths

[David Woodhouse]

From: Connor Williamson <connordw@amazon.com>

The SNP paths assume every guest_memfd PFN has a struct page and a
direct map alias.  That is not true for memory handed out by an
external guest_memfd provider that backs guest RAM with device or
carved-out physical memory (e.g. via mem=), which has no struct page.

Relax the RMP helpers in arch/x86/virt/svm/sev.c:

 - psmash(): drop the pfn_valid() gate.  PSMASH operates on the RMP
   entry for a physical address and touches neither the direct map nor
   struct page.

 - adjust_direct_map(): return 0 instead of -EINVAL for a page-less
   PFN, which has no direct map alias to split.  Checked before the 2M
   range test, which would otherwise reject the span.

 - __snp_leak_pages(): compute the struct page per iteration and skip
   list insertion when it is NULL; a page-less PFN cannot be chained
   onto the leaked pages list.

And the KVM SNP gmem paths in arch/x86/kvm/svm/sev.c:

 - sev_gmem_map_pfn()/sev_gmem_unmap_pfn() fall back to memremap() for
   the launch-update copy and the CPUID error read-back in
   sev_gmem_post_populate().  The destination PFN is mapped before the
   atomic kmap_local_page() of the source, since memremap() may sleep.

 - sev_clflush_pfn() falls back to memremap() for the cache flush in
   sev_gmem_invalidate().

No functional change for page-backed PFNs.

Signed-off-by: Connor Williamson <connordw@amazon.com>
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 arch/x86/kvm/svm/sev.c  | 83 +++++++++++++++++++++++++++++++++++------
 arch/x86/virt/svm/sev.c | 27 ++++++++++----
 2 files changed, 90 insertions(+), 20 deletions(-)

diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index 427229347876..2be9e3b80d85 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -12,6 +12,7 @@
 #include <linux/kvm_host.h>
 #include <linux/kernel.h>
 #include <linux/highmem.h>
+#include <linux/io.h>
 #include <linux/psp.h>
 #include <linux/psp-sev.h>
 #include <linux/pagemap.h>
@@ -2321,6 +2322,30 @@ struct sev_gmem_populate_args {
 	int fw_error;
 };
 
+/*
+ * Map a guest_memfd PFN for CPU access.  A PFN provided by an external
+ * guest_memfd provider may have no struct page, so fall back to memremap()
+ * for those.  Pair each call with sev_gmem_unmap_pfn().
+ */
+static void *sev_gmem_map_pfn(kvm_pfn_t pfn)
+{
+	if (pfn_valid(pfn))
+		return kmap_local_pfn(pfn);
+
+	return memremap(pfn_to_hpa(pfn), PAGE_SIZE, MEMREMAP_WB);
+}
+
+static void sev_gmem_unmap_pfn(kvm_pfn_t pfn, void *vaddr)
+{
+	if (!vaddr)
+		return;
+
+	if (pfn_valid(pfn))
+		kunmap_local(vaddr);
+	else
+		memunmap(vaddr);
+}
+
 static int sev_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
 				  struct page *src_page, void *opaque)
 {
@@ -2343,13 +2368,23 @@ static int sev_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
 	}
 
 	if (src_page) {
-		void *src_vaddr = kmap_local_page(src_page);
-		void *dst_vaddr = kmap_local_pfn(pfn);
+		void *dst_vaddr = sev_gmem_map_pfn(pfn);
+		void *src_vaddr;
 
-		memcpy(dst_vaddr, src_vaddr, PAGE_SIZE);
+		if (!dst_vaddr) {
+			ret = -ENOMEM;
+			goto out;
+		}
 
-		kunmap_local(dst_vaddr);
+		/*
+		 * Map the destination (which may be page-less and thus sleep in
+		 * memremap()) before the atomic kmap_local_page() of the source.
+		 */
+		src_vaddr = kmap_local_page(src_page);
+		memcpy(dst_vaddr, src_vaddr, PAGE_SIZE);
 		kunmap_local(src_vaddr);
+
+		sev_gmem_unmap_pfn(pfn, dst_vaddr);
 	}
 
 	ret = rmp_make_private(pfn, gfn << PAGE_SHIFT, PG_LEVEL_4K,
@@ -2379,14 +2414,17 @@ static int sev_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
 	if (ret && !snp_page_reclaim(kvm, pfn) &&
 	    sev_populate_args->type == KVM_SEV_SNP_PAGE_TYPE_CPUID &&
 	    sev_populate_args->fw_error == SEV_RET_INVALID_PARAM) {
-		void *src_vaddr = kmap_local_page(src_page);
-		void *dst_vaddr = kmap_local_pfn(pfn);
+		void *dst_vaddr = sev_gmem_map_pfn(pfn);
+		void *src_vaddr;
 
-		memcpy(src_vaddr, dst_vaddr, PAGE_SIZE);
-		set_page_dirty(src_page);
+		if (dst_vaddr) {
+			src_vaddr = kmap_local_page(src_page);
+			memcpy(src_vaddr, dst_vaddr, PAGE_SIZE);
+			set_page_dirty(src_page);
+			kunmap_local(src_vaddr);
 
-		kunmap_local(dst_vaddr);
-		kunmap_local(src_vaddr);
+			sev_gmem_unmap_pfn(pfn, dst_vaddr);
+		}
 	}
 
 out:
@@ -5117,6 +5155,28 @@ int sev_gmem_prepare(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order)
 	return 0;
 }
 
+/*
+ * Flush the CPU caches for a guest_memfd PFN.  A PFN from an external
+ * guest_memfd provider may have no direct map alias, so fall back to a
+ * temporary memremap() mapping for the cache flush.
+ */
+static void sev_clflush_pfn(kvm_pfn_t pfn, size_t size)
+{
+	void *va;
+
+	if (pfn_valid(pfn)) {
+		clflush_cache_range(__va(pfn_to_hpa(pfn)), size);
+		return;
+	}
+
+	va = memremap(pfn_to_hpa(pfn), size, MEMREMAP_WB);
+	if (WARN_ON_ONCE(!va))
+		return;
+
+	clflush_cache_range(va, size);
+	memunmap(va);
+}
+
 void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end)
 {
 	kvm_pfn_t pfn;
@@ -5172,8 +5232,7 @@ void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end)
 		 * cache entries for these pages before free'ing them back to
 		 * the host.
 		 */
-		clflush_cache_range(__va(pfn_to_hpa(pfn)),
-				    use_2m_update ? PMD_SIZE : PAGE_SIZE);
+		sev_clflush_pfn(pfn, use_2m_update ? PMD_SIZE : PAGE_SIZE);
 next_pfn:
 		pfn += use_2m_update ? PTRS_PER_PMD : 1;
 		cond_resched();
diff --git a/arch/x86/virt/svm/sev.c b/arch/x86/virt/svm/sev.c
index 8bcdce98f6dc..3623b52fbb11 100644
--- a/arch/x86/virt/svm/sev.c
+++ b/arch/x86/virt/svm/sev.c
@@ -897,8 +897,11 @@ int psmash(u64 pfn)
 	if (!cc_platform_has(CC_ATTR_HOST_SEV_SNP))
 		return -ENODEV;
 
-	if (!pfn_valid(pfn))
-		return -EINVAL;
+	/*
+	 * Do not require a struct page.  PSMASH operates on the RMP entry for
+	 * a physical address; a valid PFN with no struct page backing (e.g.
+	 * memory handed out by an external guest_memfd provider) is fine.
+	 */
 
 	/* Binutils version 2.36 supports the PSMASH mnemonic. */
 	asm volatile(".byte 0xF3, 0x0F, 0x01, 0xFF"
@@ -953,8 +956,13 @@ static int adjust_direct_map(u64 pfn, int rmp_level)
 	if (WARN_ON_ONCE(rmp_level > PG_LEVEL_2M))
 		return -EINVAL;
 
+	/*
+	 * A PFN with no struct page has no direct map alias to split, so
+	 * there is nothing to do.  This is checked before the 2M range test
+	 * below, which would otherwise reject the whole span.
+	 */
 	if (!pfn_valid(pfn))
-		return -EINVAL;
+		return 0;
 
 	if (rmp_level == PG_LEVEL_2M &&
 	    (!IS_ALIGNED(pfn, PTRS_PER_PMD) || !pfn_valid(pfn + PTRS_PER_PMD - 1)))
@@ -1060,32 +1068,35 @@ EXPORT_SYMBOL_GPL(rmp_make_shared);
 
 void __snp_leak_pages(u64 pfn, unsigned int npages, bool dump_rmp)
 {
-	struct page *page = pfn_to_page(pfn);
-
 	pr_warn("Leaking PFN range 0x%llx-0x%llx\n", pfn, pfn + npages);
 
 	spin_lock(&snp_leaked_pages_list_lock);
 	while (npages--) {
+		struct page *page = pfn_valid(pfn) ? pfn_to_page(pfn) : NULL;
 
 		/*
 		 * Reuse the page's buddy list for chaining into the leaked
 		 * pages list. This page should not be on a free list currently
 		 * and is also unsafe to be added to a free list.
+		 *
+		 * A page-less PFN (e.g. memory backed by an external
+		 * guest_memfd provider) has no struct page to chain, so it is
+		 * only accounted and, optionally, its RMP entry dumped.
 		 */
-		if (likely(!PageCompound(page)) ||
+		if (page &&
+		    (likely(!PageCompound(page)) ||
 
 			/*
 			 * Skip inserting tail pages of compound page as
 			 * page->buddy_list of tail pages is not usable.
 			 */
-		    (PageHead(page) && compound_nr(page) <= npages))
+		     (PageHead(page) && compound_nr(page) <= npages)))
 			list_add_tail(&page->buddy_list, &snp_leaked_pages_list);
 
 		if (dump_rmp)
 			dump_rmpentry(pfn);
 		snp_nr_leaked_pages++;
 		pfn++;
-		page++;
 	}
 	spin_unlock(&snp_leaked_pages_list_lock);
 }
-- 
2.54.0

[RFC PATCH 04/11] KVM: guest_memfd: Introduce guest memory ops and route native gmem through them

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

Guest_memfd today has exactly one backing implementation, hard-wired
into virt/kvm/guest_memfd.c.  To let other subsystems supply guest
memory from an alternate source -- for example a physically contiguous
range without struct page, or memory shared with a peer device --
lift the operations into a small vtable and route the built-in gmem
through it.

Introduce struct kvm_gmem_ops in <linux/kvm_host.h>: bind / unbind /
get_pfn / populate / invalidate for the memory-side operations, plus
release / mmap / fallocate for the file-level ones.  Every file that
can back guest memory is created with one shared struct
file_operations, kvm_gmem_fops, and file->private_data points at a
struct kvm_gmem_backing whose ->ops field is the per-instance vtable.
This mirrors the dma-buf / socket / iommufd pattern: a canonical fops
identifies the file (is_kvm_gmem_file() is a plain fops-equality
check), and per-instance behaviour is reached via private_data.

Every callback on kvm_gmem_fops -- release, mmap, fallocate, and the
memory-side dispatchers already present -- reaches through the ops
table, so foreign implementations get a private hook for each and
KVM code never has to distinguish "native vs foreign".  Native gmem
becomes one implementation of the ops (kvm_gmem_native_ops), with the
current release/mmap/fallocate bodies moved into
kvm_gmem_native_release/_mmap/_fallocate.  struct gmem_file gains a
struct kvm_gmem_backing as its first member so file->private_data can
be reinterpreted uniformly across implementations.

.populate is deliberately left NULL for the built-in path;
kvm_gmem_populate() keeps an inline fast path because the current
folio-locked + attribute-checked shape of post_populate() does not fit
the ops signature cleanly.  Wiring that flow through ops is a separate
cleanup.  .invalidate is likewise NULL for the built-in path, because
its invalidations run through inode->i_mapping (truncate, hwpoison,
free_folio) rather than an out-of-band notification.

kvm_gmem_unbind() gains a struct kvm * parameter so the dispatcher can
pass it to ops->unbind() without keeping a duplicate copy anywhere.
The three existing callers in kvm_main.c already have @kvm in scope.

Based on a proof-of-concept by Connor Williamson <connordw@amazon.com>.

Signed-off-by: David Woodhouse (Kiro) <dwmw@amazon.co.uk>
---
 include/linux/kvm_host.h |  82 ++++++++++
 virt/kvm/guest_memfd.c   | 338 +++++++++++++++++++++++++++++++--------
 virt/kvm/kvm_main.c      |   6 +-
 virt/kvm/kvm_mm.h        |   4 +-
 4 files changed, 359 insertions(+), 71 deletions(-)

diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index ab8cfaec82d3..b33d2f475042 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -621,6 +621,88 @@ static inline bool kvm_slot_has_gmem(const struct kvm_memory_slot *slot)
 	return slot && (slot->flags & KVM_MEM_GUEST_MEMFD);
 }
 
+#ifdef CONFIG_KVM_GUEST_MEMFD
+/*
+ * struct kvm_gmem_ops - vtable a guest_memfd backing implementation exposes.
+ *
+ * Every file that can back KVM guest memory -- KVM's own guest_memfd (see
+ * kvm_gmem_create()) and any other subsystem opting in -- is created with
+ * the shared kvm_gmem_fops (see is_kvm_gmem_file()).  file->private_data on
+ * such a file points at a struct kvm_gmem_backing whose ->ops field is the
+ * per-instance vtable below.  KVM never looks any further; the rest of the
+ * per-file state (allocator, refcount, private xarray, ...) is up to the
+ * implementation and is reached via container_of() from the backing.
+ *
+ * The memory-side callbacks (bind, unbind, get_pfn, populate, invalidate)
+ * are the reason the interface exists.  The file-level callbacks (release,
+ * mmap, fallocate, ...) are dispatched by kvm_gmem_fops so that every
+ * implementation gets a private hook -- see the dispatchers in
+ * virt/kvm/guest_memfd.c.
+ *
+ * get_pfn() contract:
+ *  - *pfn is the host PFN backing @gfn.  It need not have a struct page.
+ *  - *max_order is the largest order KVM may use to map starting at @gfn.
+ *    The implementation MUST clamp it to the physical contiguity and the
+ *    HPA/GPA alignment of the backing, so KVM never builds a huge SPTE that
+ *    spans discontiguous physical memory.
+ *  - If *page is non-NULL on return the caller owns that page reference and
+ *    is responsible for put_page() after use.  If *page is left NULL the PFN
+ *    is treated as non-refcounted, and its lifetime is owned by the
+ *    implementation across bind()/unbind().
+ *
+ * Memory intended to back guest RAM MUST be reported as E820_TYPE_RAM by the
+ * host so KVM maps it write-back (and applies the memory-encryption bit on
+ * encrypted hosts) rather than treating it as MMIO.
+ */
+struct kvm_gmem_ops {
+	/* Memory operations. */
+	int (*bind)(struct file *file, struct kvm *kvm,
+		    struct kvm_memory_slot *slot, loff_t offset);
+	void (*unbind)(struct file *file, struct kvm *kvm,
+		       struct kvm_memory_slot *slot);
+	int (*get_pfn)(struct file *file, struct kvm *kvm,
+		       struct kvm_memory_slot *slot, gfn_t gfn,
+		       kvm_pfn_t *pfn, struct page **page, int *max_order);
+	int (*populate)(struct file *file, struct kvm *kvm,
+			struct kvm_memory_slot *slot, gfn_t gfn,
+			kvm_pfn_t *pfn, struct page *src_page, int order);
+	void (*invalidate)(struct file *file, struct kvm *kvm,
+			   struct kvm_memory_slot *slot,
+			   gfn_t start, gfn_t end);
+
+	/*
+	 * File-level operations, dispatched by kvm_gmem_fops.  These are
+	 * per-instance analogues of the like-named file_operations members,
+	 * with the same semantics.  A NULL entry causes the dispatcher to
+	 * fail the file-level operation with an appropriate errno.
+	 */
+	void (*release)(struct file *file);
+	int (*mmap)(struct file *file, struct vm_area_struct *vma);
+	long (*fallocate)(struct file *file, int mode, loff_t offset,
+			  loff_t len);
+	long (*ioctl)(struct file *file, unsigned int cmd, unsigned long arg);
+};
+
+/*
+ * Anchor placed at file->private_data of every guest_memfd-backing file.
+ * KVM's own gmem embeds it as the first member of struct gmem_file; other
+ * implementations do the same in their own per-fd state.  is_kvm_gmem_file()
+ * (fops-identity check) proves the reinterpretation is safe.
+ */
+struct kvm_gmem_backing {
+	const struct kvm_gmem_ops *ops;
+};
+
+/* Defined in guest_memfd.c; single canonical fops for every gmem file. */
+extern struct file_operations kvm_gmem_fops;
+
+static inline bool is_kvm_gmem_file(struct file *file)
+{
+	return file && file->f_op == &kvm_gmem_fops;
+}
+
+#endif /* CONFIG_KVM_GUEST_MEMFD */
+
 static inline bool kvm_slot_dirty_track_enabled(const struct kvm_memory_slot *slot)
 {
 	return slot->flags & KVM_MEM_LOG_DIRTY_PAGES;
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index db57c5766ab6..2b277468a12f 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -22,11 +22,22 @@ static struct vfsmount *kvm_gmem_mnt;
  * specific to its associated VM, e.g. memslots=>gmem bindings.
  */
 struct gmem_file {
+	/*
+	 * MUST be first: file->private_data points here.
+	 * is_kvm_gmem_file(file) proves this reinterpretation is safe.
+	 */
+	struct kvm_gmem_backing backing;
+
 	struct kvm *kvm;
 	struct xarray bindings;
 	struct list_head entry;
 };
 
+static inline struct gmem_file *gmem_file_of(struct file *file)
+{
+	return container_of(file->private_data, struct gmem_file, backing);
+}
+
 struct gmem_inode {
 	struct shared_policy policy;
 	struct inode vfs_inode;
@@ -296,7 +307,7 @@ static long kvm_gmem_allocate(struct inode *inode, loff_t offset, loff_t len)
 	return r;
 }
 
-static long kvm_gmem_fallocate(struct file *file, int mode, loff_t offset,
+static long kvm_gmem_native_fallocate(struct file *file, int mode, loff_t offset,
 			       loff_t len)
 {
 	int ret;
@@ -320,9 +331,10 @@ static long kvm_gmem_fallocate(struct file *file, int mode, loff_t offset,
 	return ret;
 }
 
-static int kvm_gmem_release(struct inode *inode, struct file *file)
+static void kvm_gmem_native_release(struct file *file)
 {
-	struct gmem_file *f = file->private_data;
+	struct inode *inode = file_inode(file);
+	struct gmem_file *f = gmem_file_of(file);
 	struct kvm_memory_slot *slot;
 	struct kvm *kvm = f->kvm;
 	unsigned long index;
@@ -367,7 +379,6 @@ static int kvm_gmem_release(struct inode *inode, struct file *file)
 
 	kvm_put_kvm(kvm);
 
-	return 0;
 }
 
 static inline struct file *kvm_gmem_get_file(struct kvm_memory_slot *slot)
@@ -466,7 +477,7 @@ static const struct vm_operations_struct kvm_gmem_vm_ops = {
 #endif
 };
 
-static int kvm_gmem_mmap(struct file *file, struct vm_area_struct *vma)
+static int kvm_gmem_native_mmap(struct file *file, struct vm_area_struct *vma)
 {
 	if (!kvm_gmem_supports_mmap(file_inode(file)))
 		return -ENODEV;
@@ -481,12 +492,61 @@ static int kvm_gmem_mmap(struct file *file, struct vm_area_struct *vma)
 	return 0;
 }
 
-static struct file_operations kvm_gmem_fops = {
-	.mmap		= kvm_gmem_mmap,
+/* File-op dispatchers — thin: they all go through the backing's ops table. */
+
+static int kvm_gmem_fops_release(struct inode *inode, struct file *file)
+{
+	struct kvm_gmem_backing *b = file->private_data;
+
+	if (b && b->ops && b->ops->release)
+		b->ops->release(file);
+	return 0;
+}
+
+static int kvm_gmem_fops_mmap(struct file *file, struct vm_area_struct *vma)
+{
+	struct kvm_gmem_backing *b = file->private_data;
+
+	if (!b || !b->ops || !b->ops->mmap)
+		return -ENODEV;
+	return b->ops->mmap(file, vma);
+}
+
+static long kvm_gmem_fops_fallocate(struct file *file, int mode,
+				    loff_t offset, loff_t len)
+{
+	struct kvm_gmem_backing *b = file->private_data;
+
+	if (!b || !b->ops || !b->ops->fallocate)
+		return -EOPNOTSUPP;
+	return b->ops->fallocate(file, mode, offset, len);
+}
+
+static long kvm_gmem_fops_ioctl(struct file *file, unsigned int cmd,
+				unsigned long arg)
+{
+	struct kvm_gmem_backing *b = file->private_data;
+
+	if (!b || !b->ops || !b->ops->ioctl)
+		return -ENOTTY;
+	return b->ops->ioctl(file, cmd, arg);
+}
+
+/*
+ * A single struct file_operations for every guest_memfd-backing file --
+ * KVM's built-in gmem and any subsystem-supplied backing alike.  All
+ * callbacks that can vary per implementation dispatch through the
+ * kvm_gmem_backing at file->private_data.
+ */
+struct file_operations kvm_gmem_fops = {
 	.open		= generic_file_open,
-	.release	= kvm_gmem_release,
-	.fallocate	= kvm_gmem_fallocate,
+	.release	= kvm_gmem_fops_release,
+	.mmap		= kvm_gmem_fops_mmap,
+	.fallocate	= kvm_gmem_fops_fallocate,
+	.unlocked_ioctl	= kvm_gmem_fops_ioctl,
+	.compat_ioctl	= kvm_gmem_fops_ioctl,
 };
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_gmem_fops);
 
 static int kvm_gmem_migrate_folio(struct address_space *mapping,
 				  struct folio *dst, struct folio *src,
@@ -557,6 +617,43 @@ bool __weak kvm_arch_supports_gmem_init_shared(struct kvm *kvm)
 	return true;
 }
 
+static int  kvm_gmem_native_bind(struct file *file, struct kvm *kvm,
+				 struct kvm_memory_slot *slot, loff_t offset);
+static void kvm_gmem_native_unbind(struct file *slot_file, struct kvm *kvm,
+				   struct kvm_memory_slot *slot);
+static int  kvm_gmem_native_get_pfn(struct file *file, struct kvm *kvm,
+				    struct kvm_memory_slot *slot, gfn_t gfn,
+				    kvm_pfn_t *pfn, struct page **page,
+				    int *max_order);
+static void kvm_gmem_native_release(struct file *file);
+static int  kvm_gmem_native_mmap(struct file *file,
+				 struct vm_area_struct *vma);
+static long kvm_gmem_native_fallocate(struct file *file, int mode,
+				      loff_t offset, loff_t len);
+
+static const struct kvm_gmem_ops kvm_gmem_native_ops = {
+	/* Memory operations. */
+	.bind      = kvm_gmem_native_bind,
+	.unbind    = kvm_gmem_native_unbind,
+	.get_pfn   = kvm_gmem_native_get_pfn,
+	/*
+	 * .populate is deliberately NULL: kvm_gmem_populate() has an inline
+	 * fast path that drives post_populate() with the folio locked and
+	 * the memory-attribute check in the right place; that flow doesn't
+	 * fit the current ops signature cleanly.  Wiring it through ops
+	 * is a separate cleanup.
+	 *
+	 * .invalidate is deliberately NULL: invalidations for the built-in
+	 * implementation flow through inode->i_mapping (truncate, hwpoison,
+	 * free_folio) and don't need an out-of-band callback from KVM.
+	 */
+
+	/* File-level operations (dispatched by kvm_gmem_fops). */
+	.release   = kvm_gmem_native_release,
+	.mmap      = kvm_gmem_native_mmap,
+	.fallocate = kvm_gmem_native_fallocate,
+};
+
 static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags)
 {
 	static const char *name = "[kvm-gmem]";
@@ -605,7 +702,8 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags)
 	}
 
 	file->f_flags |= O_LARGEFILE;
-	file->private_data = f;
+	file->private_data = &f->backing;
+	f->backing.ops = &kvm_gmem_native_ops;
 
 	kvm_get_kvm(kvm);
 	f->kvm = kvm;
@@ -640,34 +738,27 @@ int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args)
 	return __kvm_gmem_create(kvm, size, flags);
 }
 
-int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot,
-		  unsigned int fd, uoff_t offset)
+static const struct kvm_gmem_ops *kvm_gmem_get_ops(struct file *file)
+{
+	if (!is_kvm_gmem_file(file))
+		return NULL;
+	return ((struct kvm_gmem_backing *)file->private_data)->ops;
+}
+
+static int kvm_gmem_native_bind(struct file *file, struct kvm *kvm,
+				struct kvm_memory_slot *slot, loff_t offset)
 {
 	uoff_t size = slot->npages << PAGE_SHIFT;
+	struct gmem_file *f = gmem_file_of(file);
+	struct inode *inode = file_inode(file);
 	unsigned long start, end;
-	struct gmem_file *f;
-	struct inode *inode;
-	struct file *file;
 	int r = -EINVAL;
 
-	BUILD_BUG_ON(sizeof(gpa_t) != sizeof(offset));
-	BUILD_BUG_ON(sizeof(gfn_t) != sizeof(slot->gmem.pgoff));
-
-	file = fget(fd);
-	if (!file)
-		return -EBADF;
-
-	if (file->f_op != &kvm_gmem_fops)
-		goto err;
-
-	f = file->private_data;
 	if (f->kvm != kvm)
-		goto err;
-
-	inode = file_inode(file);
+		return -EINVAL;
 
-	if (!PAGE_ALIGNED(offset) || offset + size > i_size_read(inode))
-		goto err;
+	if (offset < 0 || !PAGE_ALIGNED(offset) || offset + size > i_size_read(inode))
+		return -EINVAL;
 
 	filemap_invalidate_lock(inode->i_mapping);
 
@@ -677,14 +768,13 @@ int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot,
 	if (!xa_empty(&f->bindings) &&
 	    xa_find(&f->bindings, &start, end - 1, XA_PRESENT)) {
 		r = -EEXIST;
-		filemap_invalidate_unlock(inode->i_mapping);
-		goto err;
+		goto out;
 	}
 
 	/*
-	 * memslots of flag KVM_MEM_GUEST_MEMFD are immutable to change, so
-	 * kvm_gmem_bind() must occur on a new memslot.  Because the memslot
-	 * is not visible yet, kvm_gmem_get_pfn() is guaranteed to see the file.
+	 * memslots of flag KVM_MEM_GUEST_MEMFD are immutable, so bind must
+	 * occur on a new memslot.  Because the memslot is not visible yet,
+	 * kvm_gmem_get_pfn() is guaranteed to see the file.
 	 */
 	WRITE_ONCE(slot->gmem.file, file);
 	slot->gmem.pgoff = start;
@@ -692,15 +782,51 @@ int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot,
 		slot->flags |= KVM_MEMSLOT_GMEM_ONLY;
 
 	xa_store_range(&f->bindings, start, end - 1, slot, GFP_KERNEL);
+	r = 0;
+out:
 	filemap_invalidate_unlock(inode->i_mapping);
+	return r;
+}
+
+int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot,
+		  unsigned int fd, uoff_t offset)
+{
+	const struct kvm_gmem_ops *ops;
+	struct file *file;
+	int r;
+
+	BUILD_BUG_ON(sizeof(gpa_t) != sizeof(offset));
+	BUILD_BUG_ON(sizeof(gfn_t) != sizeof(slot->gmem.pgoff));
+
+	file = fget(fd);
+	if (!file)
+		return -EBADF;
+
+	ops = kvm_gmem_get_ops(file);
+	if (!ops || !ops->bind || !ops->get_pfn) {
+		r = -EINVAL;
+		goto out;
+	}
+
+	if (!PAGE_ALIGNED(offset)) {
+		r = -EINVAL;
+		goto out;
+	}
 
 	/*
-	 * Drop the reference to the file, even on success.  The file pins KVM,
-	 * not the other way 'round.  Active bindings are invalidated if the
-	 * file is closed before memslots are destroyed.
+	 * On success .bind() associates the fd with @kvm and (if the
+	 * implementation needs it) records the reference that makes the fd
+	 * pin the VM.  A .bind() that returns success is trusted to have
+	 * done so; there is no separate ownership check here.
+	 */
+	r = ops->bind(file, kvm, slot, offset);
+out:
+	/*
+	 * Drop the reference to the file even on success.  The file pins
+	 * KVM, not the other way 'round: it is the implementation's job to
+	 * invalidate its bindings if the fd is closed before the memslots
+	 * are destroyed.
 	 */
-	r = 0;
-err:
 	fput(file);
 	return r;
 }
@@ -719,21 +845,15 @@ static void __kvm_gmem_unbind(struct kvm_memory_slot *slot, struct gmem_file *f)
 	WRITE_ONCE(slot->gmem.file, NULL);
 }
 
-void kvm_gmem_unbind(struct kvm_memory_slot *slot)
+static void kvm_gmem_native_unbind(struct file *slot_file, struct kvm *kvm,
+				   struct kvm_memory_slot *slot)
 {
-	/*
-	 * Nothing to do if the underlying file was _already_ closed, as
-	 * kvm_gmem_release() invalidates and nullifies all bindings.
-	 */
-	if (!slot->gmem.file)
-		return;
-
 	CLASS(gmem_get_file, file)(slot);
 
 	/*
-	 * However, if the file is _being_ closed, then the bindings need to be
-	 * removed as kvm_gmem_release() might not run until after the memslot
-	 * is freed.  Note, modifying the bindings is safe even though the file
+	 * If the file is _being_ closed, then the bindings need to be removed
+	 * as kvm_gmem_release() might not run until after the memslot is
+	 * freed.  Note, modifying the bindings is safe even though the file
 	 * is dying as kvm_gmem_release() nullifies slot->gmem.file under
 	 * slots_lock, and only puts its reference to KVM after destroying all
 	 * bindings.  I.e. reaching this point means kvm_gmem_release() hasn't
@@ -741,15 +861,41 @@ void kvm_gmem_unbind(struct kvm_memory_slot *slot)
 	 * until the caller drops slots_lock.
 	 */
 	if (!file) {
-		__kvm_gmem_unbind(slot, slot->gmem.file->private_data);
+		__kvm_gmem_unbind(slot, gmem_file_of(slot_file));
 		return;
 	}
 
 	filemap_invalidate_lock(file->f_mapping);
-	__kvm_gmem_unbind(slot, file->private_data);
+	__kvm_gmem_unbind(slot, gmem_file_of(file));
 	filemap_invalidate_unlock(file->f_mapping);
 }
 
+void kvm_gmem_unbind(struct kvm *kvm, struct kvm_memory_slot *slot)
+{
+	const struct kvm_gmem_ops *ops;
+	struct file *file;
+
+	/*
+	 * Nothing to do if the underlying file was _already_ closed, as
+	 * kvm_gmem_release() invalidates and nullifies all bindings.
+	 */
+	file = READ_ONCE(slot->gmem.file);
+	if (!file)
+		return;
+
+	ops = kvm_gmem_get_ops(file);
+	if (ops && ops->unbind)
+		ops->unbind(file, kvm, slot);
+
+	/*
+	 * .unbind() may clear slot->gmem.file itself (as the built-in gmem
+	 * ops do, under filemap_invalidate_lock, to serialise against
+	 * release/invalidate); catch the ones that don't.
+	 */
+	if (READ_ONCE(slot->gmem.file))
+		WRITE_ONCE(slot->gmem.file, NULL);
+}
+
 /* Returns a locked folio on success.  */
 static struct folio *__kvm_gmem_get_pfn(struct file *file,
 					struct kvm_memory_slot *slot,
@@ -757,7 +903,7 @@ static struct folio *__kvm_gmem_get_pfn(struct file *file,
 					int *max_order)
 {
 	struct file *slot_file = READ_ONCE(slot->gmem.file);
-	struct gmem_file *f = file->private_data;
+	struct gmem_file *f = gmem_file_of(file);
 	struct folio *folio;
 
 	if (file != slot_file) {
@@ -787,17 +933,14 @@ static struct folio *__kvm_gmem_get_pfn(struct file *file,
 	return folio;
 }
 
-int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
-		     gfn_t gfn, kvm_pfn_t *pfn, struct page **page,
-		     int *max_order)
+static int kvm_gmem_native_get_pfn(struct file *file, struct kvm *kvm,
+				   struct kvm_memory_slot *slot, gfn_t gfn,
+				   kvm_pfn_t *pfn, struct page **page,
+				   int *max_order)
 {
 	pgoff_t index = kvm_gmem_get_index(slot, gfn);
 	struct folio *folio;
-	int r = 0;
-
-	CLASS(gmem_get_file, file)(slot);
-	if (!file)
-		return -EFAULT;
+	int r;
 
 	folio = __kvm_gmem_get_pfn(file, slot, index, pfn, max_order);
 	if (IS_ERR(folio))
@@ -809,7 +952,6 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 	}
 
 	r = kvm_gmem_prepare_folio(kvm, slot, gfn, folio);
-
 	folio_unlock(folio);
 
 	if (!r)
@@ -819,6 +961,24 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 
 	return r;
 }
+
+int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
+		     gfn_t gfn, kvm_pfn_t *pfn, struct page **page,
+		     int *max_order)
+{
+	const struct kvm_gmem_ops *ops;
+
+	CLASS(gmem_get_file, file)(slot);
+	if (!file)
+		return -EFAULT;
+
+	ops = kvm_gmem_get_ops(file);
+	if (!ops || !ops->get_pfn)
+		return -EFAULT;
+
+	*page = NULL;
+	return ops->get_pfn(file, kvm, slot, gfn, pfn, page, max_order);
+}
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_gmem_get_pfn);
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_POPULATE
@@ -860,11 +1020,50 @@ static long __kvm_gmem_populate(struct kvm *kvm, struct kvm_memory_slot *slot,
 	return ret;
 }
 
+static int kvm_gmem_populate_one(const struct kvm_gmem_ops *ops,
+				      struct file *file, struct kvm *kvm,
+				      struct kvm_memory_slot *slot, gfn_t gfn,
+				      struct page *src_page,
+				      kvm_gmem_populate_cb post_populate,
+				      void *opaque)
+{
+	kvm_pfn_t pfn;
+	int ret;
+
+	/*
+	 * Resolve the PFN via the ops.  A populate() op may also copy
+	 * @src_page into the backing before the arch post_populate() step
+	 * (e.g. SNP LAUNCH_UPDATE); otherwise fall back to get_pfn().
+	 */
+	if (ops->populate)
+		ret = ops->populate(file, kvm, slot, gfn, &pfn,
+				    src_page, 0);
+	else {
+		struct page *ignored_page = NULL;
+
+		ret = ops->get_pfn(file, kvm, slot, gfn, &pfn,
+				   &ignored_page, NULL);
+		/*
+		 * Populate does not consume a refcount on the destination page.
+		 * If get_pfn() handed a refcount back through @page, drop it.
+		 * Implementations whose PFNs aren't refcounted per-call leave
+		 * @page NULL and this is a no-op.
+		 */
+		if (!ret && ignored_page)
+			put_page(ignored_page);
+	}
+	if (ret)
+		return ret;
+
+	return post_populate(kvm, gfn, pfn, src_page, opaque);
+}
+
 long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src,
 		       long npages, bool may_writeback_src,
 		       kvm_gmem_populate_cb post_populate, void *opaque)
 {
 	struct kvm_memory_slot *slot;
+	const struct kvm_gmem_ops *ops;
 	int ret = 0;
 	long i;
 
@@ -884,6 +1083,8 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src,
 	if (!file)
 		return -EFAULT;
 
+	ops = kvm_gmem_get_ops(file);
+
 	npages = min_t(ulong, slot->npages - (start_gfn - slot->base_gfn), npages);
 	for (i = 0; i < npages; i++) {
 		struct page *src_page = NULL;
@@ -906,8 +1107,13 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src,
 			}
 		}
 
-		ret = __kvm_gmem_populate(kvm, slot, file, start_gfn + i, src_page,
-					  post_populate, opaque);
+		if (ops && (ops->populate || ops->get_pfn != kvm_gmem_native_get_pfn))
+			ret = kvm_gmem_populate_one(ops, file, kvm, slot,
+						    start_gfn + i, src_page,
+						    post_populate, opaque);
+		else
+			ret = __kvm_gmem_populate(kvm, slot, file, start_gfn + i,
+						  src_page, post_populate, opaque);
 
 		if (src_page)
 			put_page(src_page);
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index e44c20c04961..1b670201fafd 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -935,7 +935,7 @@ static void kvm_destroy_dirty_bitmap(struct kvm_memory_slot *memslot)
 static void kvm_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot)
 {
 	if (slot->flags & KVM_MEM_GUEST_MEMFD)
-		kvm_gmem_unbind(slot);
+		kvm_gmem_unbind(kvm, slot);
 
 	kvm_destroy_dirty_bitmap(slot);
 
@@ -1747,7 +1747,7 @@ static void kvm_commit_memory_region(struct kvm *kvm,
 		 * flags-only changes on guest_memfd slots should be impossible.
 		 */
 		if (WARN_ON_ONCE(old->flags & KVM_MEM_GUEST_MEMFD))
-			kvm_gmem_unbind(old);
+			kvm_gmem_unbind(kvm, old);
 
 		/*
 		 * The final quirk.  Free the detached, old slot, but only its
@@ -2118,7 +2118,7 @@ static int kvm_set_memory_region(struct kvm *kvm,
 
 out_unbind:
 	if (mem->flags & KVM_MEM_GUEST_MEMFD)
-		kvm_gmem_unbind(new);
+		kvm_gmem_unbind(kvm, new);
 out:
 	kfree(new);
 	return r;
diff --git a/virt/kvm/kvm_mm.h b/virt/kvm/kvm_mm.h
index 7510ca915dd1..ea0fe74e89d4 100644
--- a/virt/kvm/kvm_mm.h
+++ b/virt/kvm/kvm_mm.h
@@ -76,7 +76,7 @@ void kvm_gmem_exit(void);
 int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args);
 int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot,
 		  unsigned int fd, uoff_t offset);
-void kvm_gmem_unbind(struct kvm_memory_slot *slot);
+void kvm_gmem_unbind(struct kvm *kvm, struct kvm_memory_slot *slot);
 #else
 static inline int kvm_gmem_init(struct module *module)
 {
@@ -91,7 +91,7 @@ static inline int kvm_gmem_bind(struct kvm *kvm,
 	return -EIO;
 }
 
-static inline void kvm_gmem_unbind(struct kvm_memory_slot *slot)
+static inline void kvm_gmem_unbind(struct kvm *kvm, struct kvm_memory_slot *slot)
 {
 	WARN_ON_ONCE(1);
 }
-- 
2.54.0

[RFC PATCH 05/11] iommufd: Look up private-interconnect phys via exporter symbols

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

The dma-buf phys-map dispatch used by iommufd's IOMMU_IOAS_MAP_FILE path
currently knows only about the VFIO PCI exporter (looked up via
symbol_get(vfio_pci_dma_buf_iommufd_map)).  Widen the dispatch so
additional exporters can advertise the same well-known symbol name
convention and have their dma-bufs mapped without an explicit iommufd
dependency in each exporter's module.

For now the dispatch tries the sample guest_memfd provider's exporter
first, then falls back to VFIO PCI, both via symbol_get.  This mirrors
the pattern the in-tree comment already flags:

	"iommufd and vfio have a circular dependency.  Future work for a
	 phys based private interconnect will remove this."

The eventual formal negotiated dma_buf_ops op that returns phys plus a
memory-type descriptor will subsume both of these lookups.  The next
patch plumbs the memory type through iommufd; a follow-up will convert
the symbol lookups to the negotiated op.

Based on a proof-of-concept by Connor Williamson <connordw@amazon.com>

Signed-off-by: David Woodhouse (Kiro) <dwmw@amazon.co.uk>
---
 drivers/iommu/iommufd/pages.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/drivers/iommu/iommufd/pages.c b/drivers/iommu/iommufd/pages.c
index 03c8379bbc34..2d4ea41460fd 100644
--- a/drivers/iommu/iommufd/pages.c
+++ b/drivers/iommu/iommufd/pages.c
@@ -1470,6 +1470,26 @@ sym_vfio_pci_dma_buf_iommufd_map(struct dma_buf_attachment *attachment,
 	if (rc != -EOPNOTSUPP)
 		return rc;
 
+	/*
+	 * Prototype: try the sample gmem provider's dma-buf exporter.  This
+	 * mirrors the vfio-pci private-interconnect hook, and (like it) is
+	 * meant to be replaced by a formal negotiated exporter op returning
+	 * phys for iommufd.
+	 */
+	{
+		extern int gmem_provider_dma_buf_iommufd_map(
+			struct dma_buf_attachment *, struct phys_vec *);
+		typeof(&gmem_provider_dma_buf_iommufd_map) gfn;
+
+		gfn = symbol_get(gmem_provider_dma_buf_iommufd_map);
+		if (gfn) {
+			rc = gfn(attachment, phys);
+			symbol_put(gmem_provider_dma_buf_iommufd_map);
+			if (rc != -EOPNOTSUPP)
+				return rc;
+		}
+	}
+
 	if (!IS_ENABLED(CONFIG_VFIO_PCI_DMABUF))
 		return -EOPNOTSUPP;
 
-- 
2.54.0

[RFC PATCH 06/11] iommufd: Plumb dma-buf memory-type (RAM vs MMIO) through the phys map

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

iommufd's dma-buf import path programmed the IOMMU unconditionally with
BATCH_MMIO (IOMMU_MMIO set, IOMMU_CACHE cleared).  This is correct for
the VFIO PCI dma-buf exporter -- its phys is PCI BAR (MMIO) memory --
but it is the wrong prot for a dma-buf whose phys is RAM: on AMD-Vi the
device DMA silently misroutes (Identify Controller against a
gmem-provider-backed target completes with status=0 but the target
region is not written).

Carry the exporter's memory type through to the fill routine:

  - Add is_cpu_ram to iopt_pages_dmabuf and pfn_reader_dmabuf.
  - Make sym_vfio_pci_dma_buf_iommufd_map() also fill *is_cpu_ram.
    VFIO PCI's exporter is MMIO (false); the sample gmem provider's
    exporter is CPU RAM (true).
  - pfn_reader_fill_dmabuf() picks BATCH_CPU_MEMORY vs BATCH_MMIO based
    on the captured kind.

The eventual formal negotiated exporter op should return phys and
memory type together in one call, replacing both symbol_get() lookups.

Signed-off-by: David Woodhouse (Kiro) <dwmw@amazon.co.uk>
---
 drivers/iommu/iommufd/io_pagetable.h |  7 ++++++
 drivers/iommu/iommufd/pages.c        | 33 +++++++++++++++++++++++-----
 2 files changed, 34 insertions(+), 6 deletions(-)

diff --git a/drivers/iommu/iommufd/io_pagetable.h b/drivers/iommu/iommufd/io_pagetable.h
index 27e3e311d395..887ed94474c7 100644
--- a/drivers/iommu/iommufd/io_pagetable.h
+++ b/drivers/iommu/iommufd/io_pagetable.h
@@ -206,6 +206,13 @@ struct iopt_pages_dmabuf {
 	/* Always PAGE_SIZE aligned */
 	unsigned long start;
 	struct list_head tracker;
+	/*
+	 * true if the exporter's phys is CPU RAM (map with IOMMU_CACHE, no
+	 * IOMMU_MMIO); false for MMIO/BAR memory (map with IOMMU_MMIO).  Set
+	 * by the exporter dispatch at map time; consulted by
+	 * pfn_reader_fill_dmabuf() to pick BATCH_CPU_MEMORY vs BATCH_MMIO.
+	 */
+	bool is_cpu_ram;
 };
 
 /*
diff --git a/drivers/iommu/iommufd/pages.c b/drivers/iommu/iommufd/pages.c
index 2d4ea41460fd..f9b2ae6d7e96 100644
--- a/drivers/iommu/iommufd/pages.c
+++ b/drivers/iommu/iommufd/pages.c
@@ -1080,6 +1080,7 @@ static int pfn_reader_user_update_pinned(struct pfn_reader_user *user,
 struct pfn_reader_dmabuf {
 	struct phys_vec phys;
 	unsigned long start_offset;
+	bool is_cpu_ram;
 };
 
 static int pfn_reader_dmabuf_init(struct pfn_reader_dmabuf *dmabuf,
@@ -1091,6 +1092,7 @@ static int pfn_reader_dmabuf_init(struct pfn_reader_dmabuf *dmabuf,
 
 	dmabuf->phys = pages->dmabuf.phys;
 	dmabuf->start_offset = pages->dmabuf.start;
+	dmabuf->is_cpu_ram = pages->dmabuf.is_cpu_ram;
 	return 0;
 }
 
@@ -1106,9 +1108,15 @@ static int pfn_reader_fill_dmabuf(struct pfn_reader_dmabuf *dmabuf,
 	 * always filled using page size aligned PFNs just like the other types.
 	 * If the dmabuf has been sliced on a sub page offset then the common
 	 * batch to domain code will adjust it before mapping to the domain.
+	 *
+	 * The exporter's memory type (CPU RAM vs MMIO/BAR) selects the batch
+	 * kind so downstream iommu_map sets IOMMU_CACHE for cache-coherent RAM
+	 * or IOMMU_MMIO for BAR memory.  The kind was captured at map time by
+	 * the exporter dispatch.
 	 */
 	batch_add_pfn_num(batch, PHYS_PFN(dmabuf->phys.paddr + start),
-			  last_index - start_index + 1, BATCH_MMIO);
+			  last_index - start_index + 1,
+			  dmabuf->is_cpu_ram ? BATCH_CPU_MEMORY : BATCH_MMIO);
 	return 0;
 }
 
@@ -1459,22 +1467,31 @@ static const struct dma_buf_attach_ops iopt_dmabuf_attach_revoke_ops = {
  * iommufd and vfio have a circular dependency. Future work for a phys
  * based private interconnect will remove this.
  */
+/*
+ * Look up the exporter's phys accessor for iommufd's private-interconnect
+ * path.  Also fills *is_cpu_ram: true if the exporter's memory is normal
+ * cache-coherent RAM (needs BATCH_CPU_MEMORY / IOMMU_CACHE), false for MMIO
+ * (needs BATCH_MMIO / IOMMU_MMIO).  This will be replaced by a formal
+ * exporter op that returns phys + memory type together.
+ */
 static int
 sym_vfio_pci_dma_buf_iommufd_map(struct dma_buf_attachment *attachment,
-				 struct phys_vec *phys)
+				 struct phys_vec *phys, bool *is_cpu_ram)
 {
 	typeof(&vfio_pci_dma_buf_iommufd_map) fn;
 	int rc;
 
 	rc = iommufd_test_dma_buf_iommufd_map(attachment, phys);
-	if (rc != -EOPNOTSUPP)
+	if (rc != -EOPNOTSUPP) {
+		*is_cpu_ram = false;	/* test hook mimics VFIO MMIO */
 		return rc;
+	}
 
 	/*
 	 * Prototype: try the sample gmem provider's dma-buf exporter.  This
 	 * mirrors the vfio-pci private-interconnect hook, and (like it) is
 	 * meant to be replaced by a formal negotiated exporter op returning
-	 * phys for iommufd.
+	 * phys + memory type.  The provider serves RAM, so mark it CPU_RAM.
 	 */
 	{
 		extern int gmem_provider_dma_buf_iommufd_map(
@@ -1485,8 +1502,10 @@ sym_vfio_pci_dma_buf_iommufd_map(struct dma_buf_attachment *attachment,
 		if (gfn) {
 			rc = gfn(attachment, phys);
 			symbol_put(gmem_provider_dma_buf_iommufd_map);
-			if (rc != -EOPNOTSUPP)
+			if (rc != -EOPNOTSUPP) {
+				*is_cpu_ram = true;
 				return rc;
+			}
 		}
 	}
 
@@ -1498,6 +1517,7 @@ sym_vfio_pci_dma_buf_iommufd_map(struct dma_buf_attachment *attachment,
 		return -EOPNOTSUPP;
 	rc = fn(attachment, phys);
 	symbol_put(vfio_pci_dma_buf_iommufd_map);
+	*is_cpu_ram = false;	/* VFIO PCI dma-buf carries BAR (MMIO) memory */
 	return rc;
 }
 
@@ -1526,7 +1546,8 @@ static int iopt_map_dmabuf(struct iommufd_ctx *ictx, struct iopt_pages *pages,
 	if (rc)
 		goto err_detach;
 
-	rc = sym_vfio_pci_dma_buf_iommufd_map(attach, &pages->dmabuf.phys);
+	rc = sym_vfio_pci_dma_buf_iommufd_map(attach, &pages->dmabuf.phys,
+					      &pages->dmabuf.is_cpu_ram);
 	if (rc)
 		goto err_unpin;
 
-- 
2.54.0

[RFC PATCH 07/11] KVM: guest_memfd: Add ops-driven page revocation

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

Add kvm_gmem_invalidate_range(): a guest_memfd backing calls it to zap
the guest secondary-MMU mapping for a gfn range so the next access
re-faults through ops->get_pfn().  Mirrors the existing internal
__kvm_gmem_invalidate_start() for a single bound slot, bracketing the
unmap with the mmu_invalidate window so a racing vCPU fault retries
rather than installing a stale mapping.

The existing ops->invalidate callback is KVM -> implementation
(cleanup); this new symbol is the implementation -> KVM path, useful
for an overcommit manager or any subsystem that reclaims pages from a
running guest.  Export it for module use and add it to the KVM
external-export allowlist.

Signed-off-by: David Woodhouse (Kiro) <dwmw@amazon.co.uk>
---
 arch/x86/kvm/Makefile    |  3 ++-
 include/linux/kvm_host.h |  9 +++++++++
 virt/kvm/guest_memfd.c   | 39 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 50 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kvm/Makefile b/arch/x86/kvm/Makefile
index 77337c37324b..cb963d6ecaab 100644
--- a/arch/x86/kvm/Makefile
+++ b/arch/x86/kvm/Makefile
@@ -63,7 +63,8 @@ exports_grep_trailer := --include='*.[ch]' -nrw $(srctree)/virt/kvm $(srctree)/a
 				-e kvm_write_track_remove_gfn \
 				-e kvm_get_kvm \
 				-e kvm_get_kvm_safe \
-				-e kvm_put_kvm
+				-e kvm_put_kvm \
+				-e kvm_gmem_invalidate_range
 
 # Force grep to emit a goofy group separator that can in turn be replaced with
 # the above newline macro (newlines in Make are a nightmare).  Note, grep only
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index b33d2f475042..04fa0cb126f6 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -701,6 +701,15 @@ static inline bool is_kvm_gmem_file(struct file *file)
 	return file && file->f_op == &kvm_gmem_fops;
 }
 
+/*
+ * Outbound (implementation -> KVM) revocation: zap the guest's secondary MMU
+ * (NPT/EPT) mapping for a gfn range so the next guest access re-faults
+ * through the ops' get_pfn().  This is the reverse of ->invalidate (which is
+ * KVM -> implementation) and is what a backing that manages its own memory
+ * (e.g. an overcommit manager) uses to reclaim a page from a running guest.
+ */
+void kvm_gmem_invalidate_range(struct kvm *kvm, gfn_t start, gfn_t end);
+
 #endif /* CONFIG_KVM_GUEST_MEMFD */
 
 static inline bool kvm_slot_dirty_track_enabled(const struct kvm_memory_slot *slot)
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 2b277468a12f..6c13742de068 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -738,6 +738,45 @@ int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args)
 	return __kvm_gmem_create(kvm, size, flags);
 }
 
+/*
+ * Outbound (implementation -> KVM) revocation: an implementation zaps the
+ * guest's secondary MMU mapping for a gfn range so the next guest access
+ * re-faults through ops->get_pfn().  Mirrors __kvm_gmem_invalidate_start()
+ * for a single bound slot, bracketing the unmap with the mmu_invalidate
+ * window so a racing vCPU fault retries rather than installing a stale
+ * mapping.
+ */
+void kvm_gmem_invalidate_range(struct kvm *kvm, gfn_t start, gfn_t end)
+{
+	struct kvm_memory_slot *slot;
+	bool flush = false;
+	int idx;
+
+	idx = srcu_read_lock(&kvm->srcu);
+
+	slot = gfn_to_memslot(kvm, start);
+	if (slot) {
+		struct kvm_gfn_range gfn_range = {
+			.slot		= slot,
+			.start		= start,
+			.end		= min(end, slot->base_gfn + slot->npages),
+			.may_block	= true,
+			.attr_filter	= KVM_FILTER_SHARED | KVM_FILTER_PRIVATE,
+		};
+
+		KVM_MMU_LOCK(kvm);
+		kvm_mmu_invalidate_start(kvm);
+		flush = kvm_mmu_unmap_gfn_range(kvm, &gfn_range);
+		if (flush)
+			kvm_flush_remote_tlbs(kvm);
+		kvm_mmu_invalidate_end(kvm);
+		KVM_MMU_UNLOCK(kvm);
+	}
+
+	srcu_read_unlock(&kvm->srcu, idx);
+}
+EXPORT_SYMBOL_GPL(kvm_gmem_invalidate_range);
+
 static const struct kvm_gmem_ops *kvm_gmem_get_ops(struct file *file)
 {
 	if (!is_kvm_gmem_file(file))
-- 
2.54.0

[RFC PATCH 08/11] samples/kvm: Add guest_memfd backing sample

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

A minimal in-tree implementation of the guest_memfd ops interface,
useful both as reference code and as the substrate for the selftests
that follow.

Features:

- Backs a slot from a fixed page-less physical range specified by the
  module parameters addr= and len=, or falls back to a CMA-allocated
  contiguous region when neither is given.  The addr=/len= mode lets a
  reserved DRAM carve-out (mem=... or an explicit e820/DT reservation)
  become KVM slot backing without any struct page.

- Per-fd mmap capability via GMEM_PROVIDER_FLAG_MMAP_CAPABLE at
  GMEM_PROVIDER_SETUP time.  An mmap-capable fd is bound as a gmem-only
  slot (KVM_MEMSLOT_GMEM_ONLY) so the host mmap and the guest fault
  path both route through this implementation.  The mmap-capable fd is
  refused for hardware-encrypted VM types (SEV, SEV-ES, SEV-SNP, TDX)
  where a host view onto private-marked pages can only fault; the
  sample enforces this at bind time.  KVM_X86_SW_PROTECTED_VM has no
  hardware encryption and is allowed.

- Page revocation via GMEM_PROVIDER_SET_PRESENT.  Marking a range
  absent tracks it in a bitmap, invokes kvm_gmem_invalidate_range() to
  zap the guest NPT/EPT, and fans dma_buf_invalidate_mappings() out to
  any exported dma-buf so iommufd tears down its mapping.  Subsequent
  get_pfn() on the range returns -EFAULT until it is restored with
  present=1.  Models overcommit-style reclaim without needing the
  platform 's real reclaim machinery.

- A dma-buf exporter (GMEM_PROVIDER_GET_DMABUF) that yields an fd
  suitable for IOMMU_IOAS_MAP_FILE.  The exporter mirrors the shape of
  drivers/vfio/pci/vfio_pci_dmabuf.c: dma_buf_attach_revocable,
  ->map_dma_buf via dma_buf_phys_vec_to_sgt, a single-range phys_vec,
  and the "well-known symbol" hook for iommufd's private-interconnect
  phys-map lookup.

The sample is layered on top of the shared kvm_gmem_fops: its fd is
created with those fops and file->private_data points at a
kvm_gmem_backing embedded at the head of its per-instance state.
Release / mmap / ioctl for the fd all run through the ops table, so
the sample owns no file_operations of its own.

Exposing kvm_gmem_fops for module use required promoting its export
from EXPORT_SYMBOL_FOR_KVM_INTERNAL to EXPORT_SYMBOL_GPL and adding
it to the KVM external-export allowlist.

SEV-SNP bind takes a psmash + rmp_make_shared sweep over the range so
a new SNP VM can transition its pages private and re-encrypt them --
harmless on non-SNP hosts, where the calls return -ENODEV.

Based on a proof-of-concept by Connor Williamson <connordw@amazon.com>.

Signed-off-by: David Woodhouse (Kiro) <dwmw@amazon.co.uk>
---
 arch/x86/kvm/Makefile       |   3 +-
 samples/Kconfig             |  18 +
 samples/Makefile            |   1 +
 samples/kvm/Makefile        |   2 +
 samples/kvm/gmem_provider.c | 651 ++++++++++++++++++++++++++++++++++++
 samples/kvm/gmem_provider.h |  53 +++
 virt/kvm/guest_memfd.c      |   2 +-
 7 files changed, 728 insertions(+), 2 deletions(-)
 create mode 100644 samples/kvm/Makefile
 create mode 100644 samples/kvm/gmem_provider.c
 create mode 100644 samples/kvm/gmem_provider.h

diff --git a/arch/x86/kvm/Makefile b/arch/x86/kvm/Makefile
index cb963d6ecaab..535fb8cc259f 100644
--- a/arch/x86/kvm/Makefile
+++ b/arch/x86/kvm/Makefile
@@ -64,7 +64,8 @@ exports_grep_trailer := --include='*.[ch]' -nrw $(srctree)/virt/kvm $(srctree)/a
 				-e kvm_get_kvm \
 				-e kvm_get_kvm_safe \
 				-e kvm_put_kvm \
-				-e kvm_gmem_invalidate_range
+				-e kvm_gmem_invalidate_range \
+				-e kvm_gmem_fops
 
 # Force grep to emit a goofy group separator that can in turn be replaced with
 # the above newline macro (newlines in Make are a nightmare).  Note, grep only
diff --git a/samples/Kconfig b/samples/Kconfig
index a75e8e78330d..3482659638b5 100644
--- a/samples/Kconfig
+++ b/samples/Kconfig
@@ -326,6 +326,24 @@ source "samples/rust/Kconfig"
 
 source "samples/damon/Kconfig"
 
+config SAMPLE_KVM_GMEM_PROVIDER
+	tristate "Build sample guest_memfd provider -- loadable module only"
+	depends on KVM_GUEST_MEMFD && X86_64 && m
+	help
+	  This builds a sample external guest_memfd provider module with two
+	  backing modes.  Loaded with addr=/len= it backs guest memory with a
+	  fixed physical range that has no struct page -- for example memory
+	  carved out of the kernel with mem= on the command line.  Without
+	  those params it falls back to a physically contiguous, page-backed
+	  region from alloc_contig_pages(), sized by the setup ioctl (easier to
+	  run, no boot param).
+
+	  It demonstrates the guest_memfd provider ABI including 2M and 1G
+	  guest mappings and, on SEV-SNP hosts, the RMP reset that lets a
+	  page-less range be re-bound to a new VM across a live update.
+
+	  If unsure, say N.
+
 endif # SAMPLES
 
 config HAVE_SAMPLE_FTRACE_DIRECT
diff --git a/samples/Makefile b/samples/Makefile
index 07641e177bd8..e85397e5e34f 100644
--- a/samples/Makefile
+++ b/samples/Makefile
@@ -37,6 +37,7 @@ obj-$(CONFIG_SAMPLE_TPS6594_PFSM)	+= pfsm/
 subdir-$(CONFIG_SAMPLE_WATCHDOG)	+= watchdog
 subdir-$(CONFIG_SAMPLE_WATCH_QUEUE)	+= watch_queue
 obj-$(CONFIG_SAMPLE_KMEMLEAK)		+= kmemleak/
+obj-$(CONFIG_SAMPLE_KVM_GMEM_PROVIDER)	+= kvm/
 obj-$(CONFIG_SAMPLE_CORESIGHT_SYSCFG)	+= coresight/
 obj-$(CONFIG_SAMPLE_FPROBE)		+= fprobe/
 obj-$(CONFIG_SAMPLES_RUST)		+= rust/
diff --git a/samples/kvm/Makefile b/samples/kvm/Makefile
new file mode 100644
index 000000000000..dcad6e53ea78
--- /dev/null
+++ b/samples/kvm/Makefile
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: GPL-2.0
+obj-$(CONFIG_SAMPLE_KVM_GMEM_PROVIDER) += gmem_provider.o
diff --git a/samples/kvm/gmem_provider.c b/samples/kvm/gmem_provider.c
new file mode 100644
index 000000000000..e30580fecd16
--- /dev/null
+++ b/samples/kvm/gmem_provider.c
@@ -0,0 +1,651 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * gmem_provider - sample external guest_memfd provider.
+ *
+ * Demonstrates the KVM guest_memfd provider ABI with two backing modes:
+ *
+ *  - External (page-less): loaded with addr=/len=, backs guest memory with a
+ *    fixed physical range that has no struct page -- e.g. memory carved out of
+ *    the kernel with mem= on the command line.  This is the case the provider
+ *    ABI exists for; get_pfn() returns bare PFNs KVM treats as non-refcounted.
+ *
+ *  - CMA fallback (page-backed): when addr=/len= are not given, allocates a
+ *    physically contiguous region via alloc_contig_pages() sized by the setup
+ *    ioctl.  Easier to run (no mem= boot param), and still exercises 2M/1G
+ *    mappings.  Not preserved across kexec/live update.
+ *
+ * On SEV-SNP hosts, bind() resets the range's RMP entries to 4K shared so a
+ * (new) SNP VM can re-encrypt it, which is what allows re-binding the range to
+ * a fresh VM across a live update.
+ *
+ * Usage:
+ *   # page-less external range:
+ *   insmod gmem_provider.ko addr=0x5D40000000 len=0x1000000
+ *   # or CMA fallback (no params); size comes from the ioctl
+ *   insmod gmem_provider.ko
+ *   fd = open("/dev/gmem_provider"); ioctl(fd, GMEM_PROVIDER_SETUP, {kvm_fd, size});
+ *   pass the returned fd + KVM_MEM_GUEST_MEMFD to KVM_SET_USER_MEMORY_REGION2.
+ */
+
+#include <linux/anon_inodes.h>
+#include <linux/bitmap.h>
+#include <linux/dma-buf.h>
+#include <linux/dma-buf-mapping.h>
+#include <linux/dma-resv.h>
+#include <linux/file.h>
+#include <linux/fs.h>
+#include <linux/gfp.h>
+#include <linux/highmem.h>
+#include <linux/io.h>
+#include <linux/kvm_host.h>
+#include <linux/miscdevice.h>
+#include <linux/mm.h>
+#include <linux/module.h>
+#include <linux/pgtable.h>
+#include <linux/slab.h>
+
+#if IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT)
+#include <asm/sev.h>
+#endif
+
+#include "gmem_provider.h"
+
+MODULE_IMPORT_NS("DMA_BUF");
+
+static unsigned long long addr;
+module_param(addr, ullong, 0444);
+MODULE_PARM_DESC(addr, "physical base of an external page-less backing region (optional)");
+
+static unsigned long long len;
+module_param(len, ullong, 0444);
+MODULE_PARM_DESC(len, "size in bytes of the external backing region (optional)");
+
+
+struct gmem_info {
+	/*
+	 * MUST be first: file->private_data points here.  is_kvm_gmem_file()
+	 * on the KVM side proves the reinterpretation is safe.
+	 */
+	struct kvm_gmem_backing backing;
+
+	/* Kept locally: kvm_gmem_ops does not carry a kvm pointer. */
+	struct kvm *kvm;
+
+	bool mmap_capable;			/* -> KVM_MEMSLOT_GMEM_ONLY at bind */
+
+	unsigned long base_pfn;
+	unsigned long npages;
+	struct page *cma_pages;			/* non-NULL if CMA-allocated */
+	gfn_t base_gfn;				/* recorded at bind, for revoke */
+	pgoff_t pgoff;				/* provider offset (pages) of the slot */
+	unsigned long *absent;			/* bitmap of currently-revoked pages */
+	struct list_head dmabufs;		/* struct gmem_dmabuf entries */
+	struct mutex dmabufs_lock;
+};
+
+static struct gmem_info *to_gmem_info(struct file *file)
+{
+	return container_of(file->private_data, struct gmem_info, backing);
+}
+
+/* Map a backing PFN for CPU access: page-backed via kmap, page-less via memremap. */
+static void *gmem_map_pfn(kvm_pfn_t pfn)
+{
+	if (pfn_valid(pfn))
+		return kmap_local_pfn(pfn);
+	return memremap(PFN_PHYS(pfn), PAGE_SIZE, MEMREMAP_WB);
+}
+
+static void gmem_unmap_pfn(kvm_pfn_t pfn, void *vaddr)
+{
+	if (!vaddr)
+		return;
+	if (pfn_valid(pfn))
+		kunmap_local(vaddr);
+	else
+		memunmap(vaddr);
+}
+
+/* Largest order KVM may map at @gfn, snapped to 4K/2M/1G. */
+static int gmem_max_order(struct gmem_info *info, gfn_t gfn, unsigned long index)
+{
+	unsigned long pfn = info->base_pfn + index;
+	unsigned long remaining = info->npages - index;
+	unsigned int pud_order = PUD_SHIFT - PAGE_SHIFT;
+	unsigned int pmd_order = PMD_SHIFT - PAGE_SHIFT;
+
+	if (IS_ALIGNED(pfn, 1UL << pud_order) &&
+	    IS_ALIGNED(gfn, 1UL << pud_order) &&
+	    remaining >= (1UL << pud_order))
+		return pud_order;
+
+	if (IS_ALIGNED(pfn, 1UL << pmd_order) &&
+	    IS_ALIGNED(gfn, 1UL << pmd_order) &&
+	    remaining >= (1UL << pmd_order))
+		return pmd_order;
+
+	return 0;
+}
+
+static int gmem_get_pfn(struct file *file, struct kvm *kvm,
+			struct kvm_memory_slot *slot, gfn_t gfn,
+			kvm_pfn_t *pfn, struct page **page, int *max_order)
+{
+	struct gmem_info *info = to_gmem_info(file);
+	pgoff_t index = gfn - slot->base_gfn + slot->gmem.pgoff;
+
+	if (index >= info->npages)
+		return -EINVAL;
+
+	/* Revoked (absent) page: behave like not-present so the fault fails. */
+	if (info->absent && test_bit(index, info->absent))
+		return -EFAULT;
+
+	*pfn = info->base_pfn + index;
+	if (max_order)
+		*max_order = gmem_max_order(info, gfn, index);
+	return 0;
+}
+
+static int gmem_populate(struct file *file, struct kvm *kvm,
+			 struct kvm_memory_slot *slot, gfn_t gfn,
+			 kvm_pfn_t *pfn, struct page *src_page, int order)
+{
+	struct gmem_info *info = to_gmem_info(file);
+	pgoff_t index = gfn - slot->base_gfn + slot->gmem.pgoff;
+
+	if (index >= info->npages)
+		return -EINVAL;
+
+	*pfn = info->base_pfn + index;
+
+	if (src_page) {
+		void *dst, *src;
+
+		/* Map dst first: memremap() may sleep, kmap_local_page() must not. */
+		dst = gmem_map_pfn(*pfn);
+		if (!dst)
+			return -ENOMEM;
+		src = kmap_local_page(src_page);
+		memcpy(dst, src, PAGE_SIZE);
+		kunmap_local(src);
+		gmem_unmap_pfn(*pfn, dst);
+	}
+	return 0;
+}
+
+static int gmem_bind(struct file *file, struct kvm *kvm,
+		     struct kvm_memory_slot *slot, loff_t offset)
+{
+	struct gmem_info *info = to_gmem_info(file);
+	unsigned long start = offset >> PAGE_SHIFT;
+
+	if (offset < 0 || !PAGE_ALIGNED(offset) ||
+	    start + slot->npages > info->npages)
+		return -EINVAL;
+
+	/*
+	 * An mmap-capable backing hands the VMM a host mapping onto pages
+	 * that a hardware-encrypted VM (SEV, SEV-ES, SEV-SNP, TDX) will mark
+	 * private in the RMP/EPT; the mmap can only ever fault on those
+	 * pages.  Refuse rather than hand the VMM a useless (and misleading)
+	 * shared view.  SW_PROTECTED_VM has no hardware encryption and is
+	 * fine.
+	 */
+#ifdef CONFIG_X86
+	if (info->mmap_capable &&
+	    (kvm->arch.vm_type == KVM_X86_SEV_VM ||
+	     kvm->arch.vm_type == KVM_X86_SEV_ES_VM ||
+	     kvm->arch.vm_type == KVM_X86_SNP_VM ||
+	     kvm->arch.vm_type == KVM_X86_TDX_VM))
+		return -EACCES;
+#endif
+
+	/* Record the binding so the revoke ioctl can translate offset -> gfn. */
+	info->base_gfn = slot->base_gfn;
+	info->pgoff = start;
+
+#if IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT)
+	/*
+	 * Reset the RMP for the range to 4K shared so a (new) SEV-SNP VM can
+	 * transition it to private and re-encrypt it.  PSMASH any 2M entries
+	 * first.  Harmless on non-SNP hosts, where these return -ENODEV.
+	 */
+	{
+		unsigned long i;
+
+		for (i = 0; i < slot->npages; i += PTRS_PER_PMD) {
+			unsigned long pfn = info->base_pfn + start + i;
+
+			if (IS_ALIGNED(pfn, PTRS_PER_PMD))
+				psmash(pfn);
+		}
+		for (i = 0; i < slot->npages; i++) {
+			unsigned long pfn = info->base_pfn + start + i;
+			int ret = rmp_make_shared(pfn, PG_LEVEL_4K);
+
+			if (ret && ret != -ENODEV)
+				pr_info_once("gmem_provider: rmp_make_shared(0x%lx) = %d\n",
+					     pfn, ret);
+		}
+	}
+#endif
+
+	/* Claim (or, on re-bind, transfer) VM ownership; pins the VM. */
+	if (info->kvm != kvm) {
+		kvm_get_kvm(kvm);
+		if (info->kvm)
+			kvm_put_kvm(info->kvm);
+		info->kvm = kvm;
+	}
+
+	/*
+	 * Record the file on the slot (KVM's outer bind no longer does this
+	 * for us) and mark the slot gmem-only if this backing serves host
+	 * accesses through its own mmap.
+	 */
+	WRITE_ONCE(slot->gmem.file, file);
+	slot->gmem.pgoff = start;
+	if (info->mmap_capable)
+		slot->flags |= KVM_MEMSLOT_GMEM_ONLY;
+
+	return 0;
+}
+
+static void gmem_unbind(struct file *file, struct kvm *kvm,
+			struct kvm_memory_slot *slot)
+{
+#if IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT)
+	struct gmem_info *info = to_gmem_info(file);
+	unsigned long start = slot->gmem.pgoff;
+	unsigned long i;
+
+	for (i = 0; i < slot->npages; i++) {
+		unsigned long pfn = info->base_pfn + start + i;
+
+		rmp_make_shared(pfn, PG_LEVEL_4K);
+	}
+#endif
+}
+
+static void gmem_release(struct file *file);
+static int gmem_mmap(struct file *file, struct vm_area_struct *vma);
+static long gmem_fd_ioctl(struct file *file, unsigned int cmd, unsigned long arg);
+
+static const struct kvm_gmem_ops gmem_ops = {
+	.bind		= gmem_bind,
+	.unbind		= gmem_unbind,
+	.get_pfn	= gmem_get_pfn,
+	.populate	= gmem_populate,
+	.release	= gmem_release,
+	.mmap		= gmem_mmap,
+	.ioctl		= gmem_fd_ioctl,
+};
+
+static void gmem_release(struct file *file)
+{
+	struct gmem_info *info = to_gmem_info(file);
+
+	if (info->kvm)
+		kvm_put_kvm(info->kvm);
+	if (info->cma_pages)
+		free_contig_range(info->base_pfn, info->npages);
+	bitmap_free(info->absent);
+	kfree(info);
+}
+
+static int gmem_mmap(struct file *file, struct vm_area_struct *vma)
+{
+	struct gmem_info *info = to_gmem_info(file);
+	unsigned long npages = vma_pages(vma);
+
+	if (vma->vm_pgoff + npages > info->npages)
+		return -EINVAL;
+
+	/* Page-less backing: map raw PFNs, not folios. */
+	vm_flags_set(vma, VM_PFNMAP | VM_IO | VM_DONTEXPAND | VM_DONTDUMP);
+	return remap_pfn_range(vma, vma->vm_start, info->base_pfn + vma->vm_pgoff,
+			       npages << PAGE_SHIFT, vma->vm_page_prot);
+}
+
+/*
+ * Dynamic dma-buf exporter over the provider's backing.
+ *
+ * Follows the same shape as drivers/vfio/pci/vfio_pci_dmabuf.c: a per-dmabuf
+ * priv holding a phys_vec, a revocable dynamic attach, and a "private
+ * interconnect" symbol iommufd looks up to fetch phys directly (instead of
+ * mapping through the DMA API).
+ *
+ * A revoke on the provider (SET_PRESENT present=0) fans out to every exported
+ * dma-buf via dma_buf_invalidate_mappings(), so iommufd (which registered a
+ * revocable importer) tears down the IOMMU mapping alongside KVM's NPT zap.
+ */
+struct gmem_dmabuf {
+	struct dma_buf *dmabuf;
+	struct gmem_info *info;
+	struct file *provider_file;		/* holds info alive */
+	struct list_head list;			/* info->dmabufs */
+	struct phys_vec phys;			/* single contiguous range */
+	struct kref kref;
+	struct completion comp;
+	bool revoked;
+};
+
+static int gmem_dma_buf_attach(struct dma_buf *dmabuf,
+			       struct dma_buf_attachment *attach)
+{
+	struct gmem_dmabuf *priv = dmabuf->priv;
+
+	if (!attach->peer2peer)
+		return -EOPNOTSUPP;
+	if (priv->revoked)
+		return -ENODEV;
+	if (!dma_buf_attach_revocable(attach))
+		return -EOPNOTSUPP;
+	return 0;
+}
+
+static void gmem_dma_buf_done(struct kref *kref)
+{
+	struct gmem_dmabuf *priv = container_of(kref, struct gmem_dmabuf, kref);
+
+	complete(&priv->comp);
+}
+
+static struct sg_table *gmem_dma_buf_map(struct dma_buf_attachment *attach,
+					 enum dma_data_direction dir)
+{
+	struct gmem_dmabuf *priv = attach->dmabuf->priv;
+	struct sg_table *sgt;
+
+	dma_resv_assert_held(priv->dmabuf->resv);
+	if (priv->revoked)
+		return ERR_PTR(-ENODEV);
+
+	/* RAM, not P2P MMIO: no p2pdma_provider. */
+	sgt = dma_buf_phys_vec_to_sgt(attach, NULL, &priv->phys, 1,
+				      priv->phys.len, dir);
+	if (IS_ERR(sgt))
+		return sgt;
+
+	kref_get(&priv->kref);
+	return sgt;
+}
+
+static void gmem_dma_buf_unmap(struct dma_buf_attachment *attach,
+			       struct sg_table *sgt,
+			       enum dma_data_direction dir)
+{
+	struct gmem_dmabuf *priv = attach->dmabuf->priv;
+
+	dma_resv_assert_held(priv->dmabuf->resv);
+	dma_buf_free_sgt(attach, sgt, dir);
+	kref_put(&priv->kref, gmem_dma_buf_done);
+}
+
+static void gmem_dma_buf_release(struct dma_buf *dmabuf)
+{
+	struct gmem_dmabuf *priv = dmabuf->priv;
+
+	if (priv->info) {
+		mutex_lock(&priv->info->dmabufs_lock);
+		list_del_init(&priv->list);
+		mutex_unlock(&priv->info->dmabufs_lock);
+	}
+	if (priv->provider_file)
+		fput(priv->provider_file);
+	kfree(priv);
+}
+
+static const struct dma_buf_ops gmem_dma_buf_ops = {
+	.attach		= gmem_dma_buf_attach,
+	.map_dma_buf	= gmem_dma_buf_map,
+	.unmap_dma_buf	= gmem_dma_buf_unmap,
+	.release	= gmem_dma_buf_release,
+};
+
+/*
+ * Private interconnect for iommufd (mirrors vfio_pci_dma_buf_iommufd_map).
+ * Returns the single contiguous phys range for the exported region so iommufd
+ * can program the IOMMU directly, bypassing the DMA API.
+ */
+int gmem_provider_dma_buf_iommufd_map(struct dma_buf_attachment *attach,
+				      struct phys_vec *phys);
+int gmem_provider_dma_buf_iommufd_map(struct dma_buf_attachment *attach,
+				      struct phys_vec *phys)
+{
+	struct gmem_dmabuf *priv;
+
+	dma_resv_assert_held(attach->dmabuf->resv);
+	if (attach->dmabuf->ops != &gmem_dma_buf_ops)
+		return -EOPNOTSUPP;
+	priv = attach->dmabuf->priv;
+	if (priv->revoked)
+		return -ENODEV;
+	*phys = priv->phys;
+	return 0;
+}
+EXPORT_SYMBOL_FOR_MODULES(gmem_provider_dma_buf_iommufd_map, "iommufd");
+
+/* Called with info->dmabufs_lock held on the revoke path. */
+static void gmem_dma_buf_revoke_all(struct gmem_info *info)
+{
+	struct gmem_dmabuf *priv;
+
+	list_for_each_entry(priv, &info->dmabufs, list) {
+		dma_resv_lock(priv->dmabuf->resv, NULL);
+		if (!priv->revoked) {
+			priv->revoked = true;
+			dma_buf_invalidate_mappings(priv->dmabuf);
+		}
+		dma_resv_unlock(priv->dmabuf->resv);
+	}
+}
+
+static int gmem_provider_get_dmabuf(struct file *file)
+{
+	struct gmem_info *info = to_gmem_info(file);
+	DEFINE_DMA_BUF_EXPORT_INFO(exp_info);
+	struct gmem_dmabuf *priv;
+	int fd;
+
+	priv = kzalloc(sizeof(*priv), GFP_KERNEL);
+	if (!priv)
+		return -ENOMEM;
+
+	priv->info = info;
+	priv->provider_file = get_file(file);
+	priv->phys.paddr = (u64)info->base_pfn << PAGE_SHIFT;
+	priv->phys.len = (u64)info->npages << PAGE_SHIFT;
+	kref_init(&priv->kref);
+	init_completion(&priv->comp);
+	INIT_LIST_HEAD(&priv->list);
+
+	exp_info.ops = &gmem_dma_buf_ops;
+	exp_info.size = priv->phys.len;
+	exp_info.flags = O_RDWR;
+	exp_info.priv = priv;
+
+	priv->dmabuf = dma_buf_export(&exp_info);
+	if (IS_ERR(priv->dmabuf)) {
+		fd = PTR_ERR(priv->dmabuf);
+		kfree(priv);
+		return fd;
+	}
+
+	mutex_lock(&info->dmabufs_lock);
+	list_add(&priv->list, &info->dmabufs);
+	mutex_unlock(&info->dmabufs_lock);
+
+	fd = dma_buf_fd(priv->dmabuf, O_CLOEXEC);
+	if (fd < 0)
+		dma_buf_put(priv->dmabuf);
+	return fd;
+}
+
+static long gmem_fd_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
+{
+	struct gmem_info *info = to_gmem_info(file);
+	struct gmem_provider_present p;
+	unsigned long start_index, end_index;
+
+	if (cmd == GMEM_PROVIDER_GET_DMABUF)
+		return gmem_provider_get_dmabuf(file);
+
+	if (cmd != GMEM_PROVIDER_SET_PRESENT)
+		return -ENOTTY;
+	if (copy_from_user(&p, (void __user *)arg, sizeof(p)))
+		return -EFAULT;
+	if (!p.len || !PAGE_ALIGNED(p.offset) || !PAGE_ALIGNED(p.len))
+		return -EINVAL;
+
+	start_index = p.offset >> PAGE_SHIFT;
+	end_index = start_index + (p.len >> PAGE_SHIFT);
+	if (end_index > info->npages || end_index < start_index)
+		return -EINVAL;
+
+	if (p.present) {
+		/* Restore: next guest fault calls get_pfn() and re-maps. */
+		bitmap_clear(info->absent, start_index, end_index - start_index);
+	} else {
+		/* Revoke: mark absent, then zap the guest NPT/EPT for the range. */
+		bitmap_set(info->absent, start_index, end_index - start_index);
+		if (info->kvm)
+			kvm_gmem_invalidate_range(info->kvm,
+				info->base_gfn + start_index - info->pgoff,
+				info->base_gfn + end_index - info->pgoff);
+
+		/*
+		 * Fan out to iommufd (and any other dma-buf importer): mark the
+		 * exported dma-buf(s) revoked and invalidate any active mappings.
+		 * The provider stays ignorant of scratch-page policy; that lives
+		 * in the importer.
+		 */
+		mutex_lock(&info->dmabufs_lock);
+		gmem_dma_buf_revoke_all(info);
+		mutex_unlock(&info->dmabufs_lock);
+	}
+	return 0;
+}
+
+static long gmem_ctl_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
+{
+	struct gmem_provider_setup setup;
+	struct gmem_info *info;
+	struct file *kvm_file;
+	struct page *pages = NULL;
+	struct kvm *kvm;
+	unsigned long npages;
+	int fd, ret;
+
+	if (cmd != GMEM_PROVIDER_SETUP)
+		return -ENOTTY;
+
+	if (copy_from_user(&setup, (void __user *)arg, sizeof(setup)))
+		return -EFAULT;
+	if (setup.flags & ~GMEM_PROVIDER_FLAG_MMAP_CAPABLE)
+		return -EINVAL;
+
+	kvm_file = fget(setup.kvm_fd);
+	if (!kvm_file)
+		return -EBADF;
+	kvm = kvm_file->private_data;
+	if (!kvm) {
+		fput(kvm_file);
+		return -EINVAL;
+	}
+	kvm_get_kvm(kvm);
+	fput(kvm_file);
+
+	info = kzalloc(sizeof(*info), GFP_KERNEL);
+	if (!info) {
+		ret = -ENOMEM;
+		goto err_put_kvm;
+	}
+
+	if (addr && len) {
+		/* External page-less range from module params. */
+		info->base_pfn = addr >> PAGE_SHIFT;
+		info->npages = len >> PAGE_SHIFT;
+	} else {
+		/* CMA fallback: allocate a contiguous, page-backed region. */
+		if (!setup.size || !PAGE_ALIGNED(setup.size)) {
+			ret = -EINVAL;
+			goto err_free_info;
+		}
+		npages = setup.size >> PAGE_SHIFT;
+		pages = alloc_contig_pages(npages, GFP_KERNEL, numa_node_id(), NULL);
+		if (!pages) {
+			ret = -ENOMEM;
+			goto err_free_info;
+		}
+		/* Provider path skips KVM's folio-clear; zero to avoid data leak. */
+		memset(page_to_virt(pages), 0, (size_t)npages << PAGE_SHIFT);
+		info->base_pfn = page_to_pfn(pages);
+		info->npages = npages;
+		info->cma_pages = pages;
+	}
+
+	info->absent = bitmap_zalloc(info->npages, GFP_KERNEL);
+	if (!info->absent) {
+		ret = -ENOMEM;
+		goto err_free_pages;
+	}
+
+	INIT_LIST_HEAD(&info->dmabufs);
+	mutex_init(&info->dmabufs_lock);
+
+	info->backing.ops = &gmem_ops;
+	info->kvm = kvm;
+	info->mmap_capable = !!(setup.flags & GMEM_PROVIDER_FLAG_MMAP_CAPABLE);
+
+	fd = anon_inode_getfd("[gmem-provider]", &kvm_gmem_fops,
+			      &info->backing, O_RDWR | O_CLOEXEC);
+	if (fd < 0) {
+		ret = fd;
+		goto err_free_pages;
+	}
+	return fd;
+
+err_free_pages:
+	if (pages)
+		free_contig_range(page_to_pfn(pages), npages);
+err_free_info:
+	bitmap_free(info->absent);
+	kfree(info);
+err_put_kvm:
+	kvm_put_kvm(kvm);
+	return ret;
+}
+
+static const struct file_operations gmem_ctl_fops = {
+	.owner		= THIS_MODULE,
+	.unlocked_ioctl	= gmem_ctl_ioctl,
+	.compat_ioctl	= gmem_ctl_ioctl,
+};
+
+static struct miscdevice gmem_dev = {
+	.minor	= MISC_DYNAMIC_MINOR,
+	.name	= "gmem_provider",
+	.fops	= &gmem_ctl_fops,
+};
+
+static int __init gmem_provider_init(void)
+{
+	if ((addr || len) &&
+	    (!addr || !len || !PAGE_ALIGNED(addr) || !PAGE_ALIGNED(len))) {
+		pr_err("gmem_provider: addr= and len= must both be set and page aligned\n");
+		return -EINVAL;
+	}
+	return misc_register(&gmem_dev);
+}
+module_init(gmem_provider_init);
+
+static void __exit gmem_provider_exit(void)
+{
+	misc_deregister(&gmem_dev);
+}
+module_exit(gmem_provider_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("Sample guest_memfd provider (external page-less range or CMA fallback)");
diff --git a/samples/kvm/gmem_provider.h b/samples/kvm/gmem_provider.h
new file mode 100644
index 000000000000..45f1b8257f60
--- /dev/null
+++ b/samples/kvm/gmem_provider.h
@@ -0,0 +1,53 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _SAMPLES_KVM_GMEM_PROVIDER_H
+#define _SAMPLES_KVM_GMEM_PROVIDER_H
+
+#include <linux/ioctl.h>
+#include <linux/types.h>
+
+/*
+ * ioctl on /dev/gmem_provider: create a guest_memfd provider fd and return it
+ * for use with KVM_SET_USER_MEMORY_REGION2.
+ *
+ * If the module was loaded with addr=/len=, the fd is backed by that fixed
+ * page-less physical range and @size is ignored.  Otherwise the fd is backed
+ * by a @size-byte physically contiguous region from alloc_contig_pages().
+ */
+/* Flags for struct gmem_provider_setup.flags */
+#define GMEM_PROVIDER_FLAG_MMAP_CAPABLE	(1u << 0)	/* fd is mmap()-able; slot becomes gmem-only.
+								   Refused for coco VMs (SEV-SNP/TDX). */
+
+struct gmem_provider_setup {
+	__s32 kvm_fd;	/* an open KVM VM fd */
+	__u32 flags;	/* GMEM_PROVIDER_FLAG_* */
+	__u64 size;	/* CMA fallback size in bytes, page aligned */
+};
+
+#define GMEM_PROVIDER_IOCTL_BASE	'G'
+#define GMEM_PROVIDER_SETUP		_IOW(GMEM_PROVIDER_IOCTL_BASE, 1, struct gmem_provider_setup)
+
+/*
+ * ioctl on a provider fd (returned by SETUP): flip a byte range of the backing
+ * between present and absent.  Revoking (present=0) marks the range absent and
+ * zaps the guest's NPT/EPT so the next access re-faults; get_pfn() then refuses
+ * the range until it is restored (present=1).  Models overcommit page reclaim.
+ */
+struct gmem_provider_present {
+	__u64 offset;	/* byte offset into the provider region, page aligned */
+	__u64 len;	/* byte length, page aligned */
+	__u32 present;	/* 0 = revoke (absent), 1 = restore (present) */
+	__u32 pad;
+};
+
+#define GMEM_PROVIDER_SET_PRESENT	_IOW(GMEM_PROVIDER_IOCTL_BASE, 2, struct gmem_provider_present)
+
+/*
+ * ioctl on a provider fd (returned by SETUP): export the backing region as a
+ * dynamic dma-buf and return an fd for it, suitable for
+ * IOMMU_IOAS_MAP_FILE.  Revoking the region (SET_PRESENT present=0) fans out
+ * to the exported dma-buf via dma_buf_invalidate_mappings(), causing iommufd
+ * to tear down the IOMMU mapping so DMA to the reclaimed range faults.
+ */
+#define GMEM_PROVIDER_GET_DMABUF	_IO(GMEM_PROVIDER_IOCTL_BASE, 3)
+
+#endif /* _SAMPLES_KVM_GMEM_PROVIDER_H */
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 6c13742de068..2cb1b6a9d73f 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -546,7 +546,7 @@ struct file_operations kvm_gmem_fops = {
 	.unlocked_ioctl	= kvm_gmem_fops_ioctl,
 	.compat_ioctl	= kvm_gmem_fops_ioctl,
 };
-EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_gmem_fops);
+EXPORT_SYMBOL_GPL(kvm_gmem_fops);
 
 static int kvm_gmem_migrate_folio(struct address_space *mapping,
 				  struct folio *dst, struct folio *src,
-- 
2.54.0

[RFC PATCH 09/11] selftests/kvm: gmem_provider KVM-only tests

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

Three selftests that exercise the guest_memfd provider ABI purely from
the KVM side (no iommufd, no assigned devices):

  gmem_provider_test
    SEV-SNP launch + re-bind against a page-less provider region.
    Loads a 16 MiB guest at physical addr= from the module parameters,
    runs SNP_LAUNCH_START -> SNP_LAUNCH_UPDATE -> SNP_LAUNCH_FINISH ->
    KVM_CREATE_VCPU, tears the VM down, and re-binds the same provider
    fd to a fresh SNP VM.  Proves the LU-survivable path (fd persists,
    KVM refs transfer, RMP transitions clean).

  gmem_provider_hugepage_test
    Non-CoCo (SW_PROTECTED_VM) guest whose slot is backed by a
    mmap-capable provider fd.  Verifies via KVM_GET_STATS that the NPT
    for a 1 GiB, 1G-aligned slot contains at least one 1 GiB and one
    2 MiB mapping.  Exercises kvm_gmem_max_mapping_level and the
    mmap-capable = gmem-only path.

  gmem_provider_revoke_test
    GMEM_PROVIDER_SET_PRESENT round trip.  Reads a page from the guest
    (populates NPT), revokes the range, re-enters the guest to observe
    the fault (memory attribute goes absent, vcpu_run fails EFAULT),
    restores and re-enters, confirms the guest reads the original
    value.  Exercises the provider-driven kvm_gmem_provider_invalidate
    helper end-to-end.

All three assume the sample provider is loaded with an addr=/len=
page-less region; hugepage and revoke additionally require the fd to
be set up with GMEM_PROVIDER_FLAG_MMAP_CAPABLE.

Signed-off-by: David Woodhouse (Kiro) <dwmw@amazon.co.uk>
---
 tools/testing/selftests/kvm/Makefile.kvm      |   3 +
 .../kvm/x86/gmem_provider_hugepage_test.c     | 130 ++++++++++++
 .../kvm/x86/gmem_provider_revoke_test.c       | 134 ++++++++++++
 .../selftests/kvm/x86/gmem_provider_test.c    | 195 ++++++++++++++++++
 4 files changed, 462 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/x86/gmem_provider_hugepage_test.c
 create mode 100644 tools/testing/selftests/kvm/x86/gmem_provider_revoke_test.c
 create mode 100644 tools/testing/selftests/kvm/x86/gmem_provider_test.c

diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm
index d28a057fa6c2..e6e9ac9da45a 100644
--- a/tools/testing/selftests/kvm/Makefile.kvm
+++ b/tools/testing/selftests/kvm/Makefile.kvm
@@ -161,6 +161,9 @@ TEST_GEN_PROGS_x86 += rseq_test
 TEST_GEN_PROGS_x86 += steal_time
 TEST_GEN_PROGS_x86 += system_counter_offset_test
 TEST_GEN_PROGS_x86 += pre_fault_memory_test
+TEST_GEN_PROGS_x86 += x86/gmem_provider_test
+TEST_GEN_PROGS_x86 += x86/gmem_provider_hugepage_test
+TEST_GEN_PROGS_x86 += x86/gmem_provider_revoke_test
 
 # Compiled outputs used by test targets
 TEST_GEN_PROGS_EXTENDED_x86 += x86/nx_huge_pages_test
diff --git a/tools/testing/selftests/kvm/x86/gmem_provider_hugepage_test.c b/tools/testing/selftests/kvm/x86/gmem_provider_hugepage_test.c
new file mode 100644
index 000000000000..fbbef9761e64
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/gmem_provider_hugepage_test.c
@@ -0,0 +1,130 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * gmem_provider_hugepage_test - verify KVM installs 2M and 1G NPT/EPT mappings
+ * for a non-confidential guest backed entirely by the samples/kvm gmem_provider
+ * in its gmem-only (mmap) mode.
+ *
+ * The test opens the provider with GMEM_PROVIDER_FLAG_MMAP_CAPABLE at SETUP time.
+ * Load the module with a >=1G, 1G-aligned page-less range:
+ *   insmod gmem_provider.ko addr=0x5D40000000 len=0x42000000
+ *
+ * The provider fd is mmap-capable, so its slot is gmem-only: all guest faults
+ * route to the provider (no private attributes needed) and the mapping level
+ * comes from the provider's max_order. The slot's userspace_addr is a
+ * 1G-aligned mmap of the provider, so the lpage HVA-alignment check passes and
+ * 1G is permitted. Skips if the module or SW_PROTECTED VM support is absent.
+ */
+#include <fcntl.h>
+#include <errno.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <string.h>
+#include <unistd.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+
+#include <linux/sizes.h>
+
+#include "test_util.h"
+#include "kvm_util.h"
+#include "processor.h"
+
+/* Mirrors samples/kvm/gmem_provider.h */
+struct gmem_provider_setup {
+	__s32 kvm_fd;
+	__u32 flags;
+	__u64 size;
+};
+#define GMEM_PROVIDER_SETUP _IOW('G', 1, struct gmem_provider_setup)
+#define GMEM_PROVIDER_FLAG_MMAP_CAPABLE (1u << 0)
+
+#define DATA_SLOT	10
+#define DATA_GPA	(1ULL << 32)		/* 4G: 1G-aligned */
+#define DATA_SIZE	((uint64_t)SZ_1G + SZ_2M)
+
+static void guest_code(void)
+{
+	*(volatile uint64_t *)DATA_GPA = 0xdead;		/* -> 1G mapping */
+	*(volatile uint64_t *)(DATA_GPA + SZ_1G) = 0xbeef;	/* -> 2M mapping */
+	GUEST_DONE();
+}
+
+int main(void)
+{
+	struct vm_shape shape = {
+		.mode = VM_MODE_DEFAULT,
+		.type = KVM_X86_SW_PROTECTED_VM,
+	};
+	struct gmem_provider_setup setup = { .flags = GMEM_PROVIDER_FLAG_MMAP_CAPABLE };
+	struct kvm_vcpu *vcpu;
+	struct kvm_vm *vm;
+	struct ucall uc;
+	int gmem_ctl, gmem_fd, r;
+	void *resv, *hva;
+	uint64_t p4k, p2m, p1g;
+
+	TEST_REQUIRE(kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SW_PROTECTED_VM));
+
+	gmem_ctl = open("/dev/gmem_provider", O_RDWR);
+	__TEST_REQUIRE(gmem_ctl >= 0,
+		       "gmem_provider module not loaded (/dev/gmem_provider absent)");
+
+	vm = vm_create_shape_with_one_vcpu(shape, &vcpu, guest_code);
+
+	setup.kvm_fd = vm->fd;
+	setup.size = DATA_SIZE;
+	gmem_fd = ioctl(gmem_ctl, GMEM_PROVIDER_SETUP, &setup);
+	TEST_ASSERT(gmem_fd >= 0, "GMEM_PROVIDER_SETUP failed, errno %d", errno);
+
+	/*
+	 * Map the provider at a 1G-aligned host VA so the slot's userspace_addr
+	 * is 1G-congruent with the (1G-aligned) GPA and KVM permits 1G lpages.
+	 * A non-mmap-capable provider (the coco path) would fail this mmap.
+	 */
+	resv = mmap(NULL, DATA_SIZE + SZ_1G, PROT_NONE,
+		    MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE, -1, 0);
+	TEST_ASSERT(resv != MAP_FAILED, "reserve VA failed, errno %d", errno);
+	hva = (void *)(((uintptr_t)resv + SZ_1G - 1) & ~((uintptr_t)SZ_1G - 1));
+	hva = mmap(hva, DATA_SIZE, PROT_READ | PROT_WRITE,
+		   MAP_SHARED | MAP_FIXED, gmem_fd, 0);
+	TEST_ASSERT(hva != MAP_FAILED,
+		    "mmap(provider) failed, errno %d", errno);
+
+	/* gmem-only slot: the provider is the whole backing; host uses @hva. */
+	r = __vm_set_user_memory_region2(vm, DATA_SLOT, KVM_MEM_GUEST_MEMFD,
+					 DATA_GPA, DATA_SIZE, hva, gmem_fd, 0);
+	TEST_ASSERT(!r, "KVM_SET_USER_MEMORY_REGION2 failed: %d errno %d", r, errno);
+
+	/* Map only the two pages the guest touches into its page tables. */
+	virt_map(vm, DATA_GPA, DATA_GPA, 1);
+	virt_map(vm, DATA_GPA + SZ_1G, DATA_GPA + SZ_1G, 1);
+
+	vcpu_run(vcpu);
+	switch (get_ucall(vcpu, &uc)) {
+	case UCALL_DONE:
+		break;
+	case UCALL_ABORT:
+		REPORT_GUEST_ASSERT(uc);
+	default:
+		TEST_FAIL("Unexpected exit: %s",
+			  exit_reason_str(vcpu->run->exit_reason));
+	}
+
+	p4k = vm_get_stat(vm, pages_4k);
+	p2m = vm_get_stat(vm, pages_2m);
+	p1g = vm_get_stat(vm, pages_1g);
+	pr_info("gmem-only provider NPT mappings: 4K=%lu 2M=%lu 1G=%lu\n",
+		p4k, p2m, p1g);
+	/* Informational: the host mapping and guest share the same memory. */
+	pr_info("host reads at DATA_GPA via mmap: 0x%llx\n",
+		(unsigned long long)*(volatile uint64_t *)hva);
+
+	TEST_ASSERT(p1g > 0, "expected at least one 1G NPT mapping, got 0");
+	TEST_ASSERT(p2m > 0, "expected at least one 2M NPT mapping, got 0");
+
+	kvm_vm_free(vm);
+	munmap(hva, DATA_SIZE);
+	close(gmem_fd);
+	close(gmem_ctl);
+	return 0;
+}
diff --git a/tools/testing/selftests/kvm/x86/gmem_provider_revoke_test.c b/tools/testing/selftests/kvm/x86/gmem_provider_revoke_test.c
new file mode 100644
index 000000000000..7e711ea52674
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/gmem_provider_revoke_test.c
@@ -0,0 +1,134 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * gmem_provider_revoke_test - exercise provider-driven page revocation.
+ *
+ * Flips a provider-backed page between present and absent via an ioctl on the
+ * provider fd and checks the provider->KVM path: revoking zaps the guest's NPT
+ * (so the guest re-faults into an absent get_pfn and cannot read the page),
+ * restoring lets the next fault re-map it. This is the mechanism a memory
+ * overcommit manager would use to reclaim a page from a running guest.
+ *
+ * The test opens the provider with GMEM_PROVIDER_FLAG_MMAP_CAPABLE at SETUP time (gmem-only). Load the module with a
+ * backing region of at least DATA_SIZE, e.g.:
+ *   insmod gmem_provider.ko addr=0x5D40000000 len=0x1000000
+ */
+#include <fcntl.h>
+#include <errno.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <string.h>
+#include <unistd.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+
+#include "test_util.h"
+#include "kvm_util.h"
+#include "processor.h"
+
+/* Mirrors samples/kvm/gmem_provider.h */
+struct gmem_provider_setup {
+	__s32 kvm_fd;
+	__u32 flags;
+	__u64 size;
+};
+struct gmem_provider_present {
+	__u64 offset;
+	__u64 len;
+	__u32 present;
+	__u32 pad;
+};
+#define GMEM_PROVIDER_SETUP		_IOW('G', 1, struct gmem_provider_setup)
+#define GMEM_PROVIDER_FLAG_MMAP_CAPABLE	(1u << 0)
+#define GMEM_PROVIDER_SET_PRESENT	_IOW('G', 2, struct gmem_provider_present)
+
+#define DATA_SLOT	10
+#define DATA_GPA	(1ULL << 32)
+#define DATA_SIZE	0x200000ULL		/* 2 MiB region */
+#define MAGIC		0x1234abcdULL
+
+static void guest_code(void)
+{
+	*(volatile uint64_t *)DATA_GPA = MAGIC;
+	for (;;)
+		GUEST_SYNC(*(volatile uint64_t *)DATA_GPA);
+}
+
+int main(void)
+{
+	struct vm_shape shape = {
+		.mode = VM_MODE_DEFAULT,
+		.type = KVM_X86_SW_PROTECTED_VM,
+	};
+	struct gmem_provider_setup setup = { .flags = GMEM_PROVIDER_FLAG_MMAP_CAPABLE };
+	struct gmem_provider_present req;
+	struct kvm_vcpu *vcpu;
+	struct kvm_vm *vm;
+	struct ucall uc;
+	int gmem_ctl, gmem_fd, r;
+	void *hva;
+
+	TEST_REQUIRE(kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SW_PROTECTED_VM));
+
+	gmem_ctl = open("/dev/gmem_provider", O_RDWR);
+	__TEST_REQUIRE(gmem_ctl >= 0,
+		       "gmem_provider module not loaded (/dev/gmem_provider absent)");
+
+	vm = vm_create_shape_with_one_vcpu(shape, &vcpu, guest_code);
+
+	setup.kvm_fd = vm->fd;
+	setup.size = DATA_SIZE;
+	gmem_fd = ioctl(gmem_ctl, GMEM_PROVIDER_SETUP, &setup);
+	TEST_ASSERT(gmem_fd >= 0, "GMEM_PROVIDER_SETUP failed, errno %d", errno);
+
+	hva = mmap(NULL, DATA_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, gmem_fd, 0);
+	TEST_ASSERT(hva != MAP_FAILED,
+		    "mmap(provider) failed, errno %d", errno);
+
+	r = __vm_set_user_memory_region2(vm, DATA_SLOT, KVM_MEM_GUEST_MEMFD,
+					 DATA_GPA, DATA_SIZE, hva, gmem_fd, 0);
+	TEST_ASSERT(!r, "KVM_SET_USER_MEMORY_REGION2 failed: %d errno %d", r, errno);
+	virt_map(vm, DATA_GPA, DATA_GPA, 1);
+
+	/* 1) Present: guest writes MAGIC and reads it back. */
+	vcpu_run(vcpu);
+	TEST_ASSERT(get_ucall(vcpu, &uc) == UCALL_SYNC, "expected UCALL_SYNC");
+	TEST_ASSERT(uc.args[1] == MAGIC, "guest read 0x%lx, want MAGIC",
+		    (unsigned long)uc.args[1]);
+	pr_info("present: guest read 0x%llx\n", MAGIC);
+
+	/* 2) Revoke: mark absent and zap the guest NPT (provider->KVM). */
+	req = (struct gmem_provider_present){ .offset = 0, .len = 4096, .present = 0 };
+	r = ioctl(gmem_fd, GMEM_PROVIDER_SET_PRESENT, &req);
+	TEST_ASSERT(!r, "revoke ioctl failed, errno %d", errno);
+
+	/* 3) Guest re-reads -> re-fault into absent get_pfn -> must NOT see MAGIC. */
+	r = _vcpu_run(vcpu);
+	if (r) {
+		pr_info("revoked: vcpu_run failed (errno %d) -- guest faulted, as expected\n",
+			errno);
+	} else {
+		TEST_ASSERT(vcpu->run->exit_reason == KVM_EXIT_MEMORY_FAULT,
+			    "revoked: expected fault or KVM_EXIT_MEMORY_FAULT, got %u (%s)",
+			    vcpu->run->exit_reason,
+			    exit_reason_str(vcpu->run->exit_reason));
+		pr_info("revoked: KVM_EXIT_MEMORY_FAULT as expected\n");
+	}
+
+	/* 4) Restore: mark present again. */
+	req.present = 1;
+	r = ioctl(gmem_fd, GMEM_PROVIDER_SET_PRESENT, &req);
+	TEST_ASSERT(!r, "restore ioctl failed, errno %d", errno);
+
+	/* 5) Re-enter: the fault re-maps via get_pfn, guest reads MAGIC again. */
+	vcpu_run(vcpu);
+	TEST_ASSERT(get_ucall(vcpu, &uc) == UCALL_SYNC, "expected UCALL_SYNC after restore");
+	TEST_ASSERT(uc.args[1] == MAGIC, "after restore guest read 0x%lx, want MAGIC",
+		    (unsigned long)uc.args[1]);
+	pr_info("restored: guest read 0x%llx again -- revoke/restore path works\n", MAGIC);
+
+	kvm_vm_free(vm);
+	munmap(hva, DATA_SIZE);
+	close(gmem_fd);
+	close(gmem_ctl);
+	return 0;
+}
diff --git a/tools/testing/selftests/kvm/x86/gmem_provider_test.c b/tools/testing/selftests/kvm/x86/gmem_provider_test.c
new file mode 100644
index 000000000000..d7caa11616df
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/gmem_provider_test.c
@@ -0,0 +1,195 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * gmem_provider_test - exercise the samples/kvm gmem_provider module through a
+ * full SEV-SNP guest launch, then re-bind the same provider fd to a second VM
+ * (the live-update path).
+ *
+ * The provider module must be loaded first, in either mode:
+ *   insmod gmem_provider.ko addr=0x5D40000000 len=0x1000000   # page-less
+ *   insmod gmem_provider.ko                                   # CMA fallback
+ *
+ * The test skips (KSFT_SKIP) if /dev/gmem_provider or SNP support is absent.
+ *
+ * Note: with the current kvm_gmem_populate() ABI the launch source is an
+ * ordinary anonymous buffer (pinned via get_user_pages_fast); the provider's
+ * populate() copies it into the backing.  No /dev/mem mapping is needed.
+ */
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <fcntl.h>
+#include <errno.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <stdint.h>
+#include <linux/kvm.h>
+#include <asm/kvm.h>
+
+/* Mirrors samples/kvm/gmem_provider.h */
+struct gmem_provider_setup {
+	int32_t kvm_fd;
+	uint32_t pad;
+	uint64_t size;
+};
+#define GMEM_PROVIDER_SETUP _IOW('G', 1, struct gmem_provider_setup)
+
+#define KSFT_PASS 0
+#define KSFT_FAIL 1
+#define KSFT_SKIP 4
+
+#define GUEST_MEM_SIZE	(16UL * 1024 * 1024)
+#define PAGE_SIZE_4K	4096UL
+
+static int sev_ioctl(int vm_fd, int sev_fd, int cmd, void *data)
+{
+	struct kvm_sev_cmd sev_cmd = {
+		.id = cmd,
+		.data = (uint64_t)(unsigned long)data,
+		.sev_fd = sev_fd,
+	};
+	return ioctl(vm_fd, KVM_MEMORY_ENCRYPT_OP, &sev_cmd);
+}
+
+static int launch_snp_vm(int kvm_fd, int gmem_fd, void *src, int vm_num)
+{
+	int vm_fd, vcpu_fd, sev_fd, ret;
+
+	printf("[VM%d] KVM_CREATE_VM (SNP)\n", vm_num);
+	vm_fd = ioctl(kvm_fd, KVM_CREATE_VM, KVM_X86_SNP_VM);
+	if (vm_fd < 0) { perror("KVM_CREATE_VM"); return -1; }
+
+	sev_fd = open("/dev/sev", O_RDWR);
+	if (sev_fd < 0) { perror("open /dev/sev"); close(vm_fd); return -1; }
+
+	struct kvm_sev_init init = { 0 };
+	ret = sev_ioctl(vm_fd, sev_fd, KVM_SEV_INIT2, &init);
+	if (ret) { perror("KVM_SEV_INIT2"); goto out; }
+
+	struct kvm_userspace_memory_region2 region = {
+		.slot = 0,
+		.flags = KVM_MEM_GUEST_MEMFD,
+		.guest_phys_addr = 0,
+		.memory_size = GUEST_MEM_SIZE,
+		.userspace_addr = (uint64_t)(unsigned long)src,
+		.guest_memfd = gmem_fd,
+		.guest_memfd_offset = 0,
+	};
+	ret = ioctl(vm_fd, KVM_SET_USER_MEMORY_REGION2, &region);
+	if (ret) { perror("KVM_SET_USER_MEMORY_REGION2"); goto out; }
+
+	struct kvm_memory_attributes attrs = {
+		.address = 0,
+		.size = GUEST_MEM_SIZE,
+		.attributes = KVM_MEMORY_ATTRIBUTE_PRIVATE,
+	};
+	ret = ioctl(vm_fd, KVM_SET_MEMORY_ATTRIBUTES, &attrs);
+	if (ret) { perror("KVM_SET_MEMORY_ATTRIBUTES"); goto out; }
+
+	struct kvm_sev_snp_launch_start start = { .policy = 0x30000 };
+	ret = sev_ioctl(vm_fd, sev_fd, KVM_SEV_SNP_LAUNCH_START, &start);
+	if (ret) { perror("SNP_LAUNCH_START"); goto out; }
+
+	printf("[VM%d] SNP_LAUNCH_UPDATE (code + zero, %luMB)\n",
+	       vm_num, GUEST_MEM_SIZE >> 20);
+	struct kvm_sev_snp_launch_update update = {
+		.gfn_start = 0,
+		.uaddr = (uint64_t)(unsigned long)src,
+		.len = PAGE_SIZE_4K,
+		.type = KVM_SEV_SNP_PAGE_TYPE_NORMAL,
+	};
+	ret = sev_ioctl(vm_fd, sev_fd, KVM_SEV_SNP_LAUNCH_UPDATE, &update);
+	if (ret) { perror("SNP_LAUNCH_UPDATE code"); goto out; }
+
+	struct kvm_sev_snp_launch_update update_zero = {
+		.gfn_start = 1,
+		.uaddr = (uint64_t)(unsigned long)(src + PAGE_SIZE_4K),
+		.len = GUEST_MEM_SIZE - PAGE_SIZE_4K,
+		.type = KVM_SEV_SNP_PAGE_TYPE_ZERO,
+	};
+	ret = sev_ioctl(vm_fd, sev_fd, KVM_SEV_SNP_LAUNCH_UPDATE, &update_zero);
+	if (ret) { perror("SNP_LAUNCH_UPDATE zero"); goto out; }
+
+	struct kvm_sev_snp_launch_finish finish = { 0 };
+	ret = sev_ioctl(vm_fd, sev_fd, KVM_SEV_SNP_LAUNCH_FINISH, &finish);
+	if (ret) { perror("SNP_LAUNCH_FINISH"); goto out; }
+
+	vcpu_fd = ioctl(vm_fd, KVM_CREATE_VCPU, 0);
+	if (vcpu_fd < 0) { perror("KVM_CREATE_VCPU"); ret = -1; goto out; }
+
+	printf("[VM%d] *** launch succeeded ***\n", vm_num);
+	close(vcpu_fd);
+	ret = 0;
+out:
+	close(sev_fd);
+	close(vm_fd);
+	return ret;
+}
+
+int main(void)
+{
+	int kvm_fd, gmem_ctl_fd, gmem_fd, tmp_vm;
+	void *src;
+
+	setbuf(stdout, NULL);
+
+	kvm_fd = open("/dev/kvm", O_RDWR);
+	if (kvm_fd < 0) {
+		printf("SKIP: cannot open /dev/kvm (%s)\n", strerror(errno));
+		return KSFT_SKIP;
+	}
+
+	gmem_ctl_fd = open("/dev/gmem_provider", O_RDWR);
+	if (gmem_ctl_fd < 0) {
+		printf("SKIP: /dev/gmem_provider not present -- load gmem_provider.ko (%s)\n",
+		       strerror(errno));
+		return KSFT_SKIP;
+	}
+
+	/* Provider setup needs a VM fd; ownership transfers to each VM on bind. */
+	tmp_vm = ioctl(kvm_fd, KVM_CREATE_VM, KVM_X86_SNP_VM);
+	if (tmp_vm < 0) {
+		printf("SKIP: cannot create SNP VM -- host not SNP-capable? (%s)\n",
+		       strerror(errno));
+		return KSFT_SKIP;
+	}
+
+	struct gmem_provider_setup setup = {
+		.kvm_fd = tmp_vm,
+		.size = GUEST_MEM_SIZE,
+	};
+	gmem_fd = ioctl(gmem_ctl_fd, GMEM_PROVIDER_SETUP, &setup);
+	if (gmem_fd < 0) {
+		perror("GMEM_PROVIDER_SETUP");
+		return KSFT_FAIL;
+	}
+	close(tmp_vm);
+	printf("provider gmem_fd = %d (persists across VMs)\n", gmem_fd);
+
+	/* Launch source: ordinary anonymous memory holding a HLT at gfn 0. */
+	src = mmap(NULL, GUEST_MEM_SIZE, PROT_READ | PROT_WRITE,
+		   MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+	if (src == MAP_FAILED) { perror("mmap src"); return KSFT_FAIL; }
+	memset(src, 0, GUEST_MEM_SIZE);
+	((uint8_t *)src)[0] = 0xf4; /* HLT */
+
+	if (launch_snp_vm(kvm_fd, gmem_fd, src, 1) != 0) {
+		printf("FAIL: VM1 launch failed\n");
+		return KSFT_FAIL;
+	}
+
+	usleep(100000);
+
+	/* Re-bind the SAME provider fd to a fresh VM (live-update path). */
+	if (launch_snp_vm(kvm_fd, gmem_fd, src, 2) != 0) {
+		printf("FAIL: VM2 re-bind launch failed\n");
+		return KSFT_FAIL;
+	}
+
+	printf("PASS: page-less/provider-backed SNP launch + re-bind succeeded\n");
+	munmap(src, GUEST_MEM_SIZE);
+	close(gmem_fd);
+	close(gmem_ctl_fd);
+	close(kvm_fd);
+	return KSFT_PASS;
+}
-- 
2.54.0

[RFC PATCH 10/11] selftests/kvm: gmem_provider iommufd tests

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

Three selftests that exercise the guest_memfd provider ABI from the
iommufd side, going progressively closer to a real assigned-device
setup:

  gmem_provider_iommufd_test
    Dual-map.  A KVM memslot and an iommufd IOAS are both backed by the
    same provider dma-buf.  The guest writes a magic pattern into the
    slot; the host mmap of the same provider fd sees the guest's write;
    the IOAS also holds a mapping of the same backing.  Proves KVM and
    iommufd converge on the same provider phys via two different fds
    (the KVM guest_memfd fd and the dma-buf fd).

  gmem_provider_vfio_test
    A real vfio-pci device is bound to iommufd and attached to an IOAS
    that has the provider region mapped.  Confirms the attach path
    accepts a dma-buf-backed IOAS without complaint; does not issue
    device DMA.  Requires an unused PCI device already bound to
    vfio-pci and exposed as /dev/vfio/devices/vfio0 (overridable via
    GMEM_VFIO_CDEV).

  gmem_provider_nvme_dma_test
    Full round trip.  Binds an NVMe controller to vfio-pci, attaches
    it to an IOAS with the provider region mapped at IOVA_DST,
    initialises NVMe admin queues (also in the IOAS via anonymous
    pages), submits Identify Controller with PRP1=IOVA_DST, and
    verifies the 4 KiB response via the host mmap of the same
    provider fd (VID/SSVID/MN check).  Then revokes the target range
    with GMEM_PROVIDER_SET_PRESENT and re-submits Identify to the same
    IOVA: iommufd's iopt_revoke_notify tears down the mapping and the
    device DMA either faults at the IOMMU (visible as IO_PAGE_FAULT in
    dmesg on AMD-Vi / equivalent on VT-d, SMMUv3) or completes with an
    NVMe error; either way the provider region is not overwritten.
    Restores and confirms fresh mappings succeed.

Requires an EBS/NVMe scratch device on the test host bound to vfio-pci;
the tests SKIP cleanly if no such device is available.

Signed-off-by: David Woodhouse (Kiro) <dwmw@amazon.co.uk>
---
 tools/testing/selftests/kvm/Makefile.kvm      |   9 +-
 .../kvm/x86/gmem_provider_iommufd_test.c      | 150 +++++++
 .../kvm/x86/gmem_provider_nvme_dma_test.c     | 365 ++++++++++++++++++
 .../kvm/x86/gmem_provider_vfio_test.c         | 134 +++++++
 4 files changed, 655 insertions(+), 3 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/x86/gmem_provider_iommufd_test.c
 create mode 100644 tools/testing/selftests/kvm/x86/gmem_provider_nvme_dma_test.c
 create mode 100644 tools/testing/selftests/kvm/x86/gmem_provider_vfio_test.c

diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm
index e6e9ac9da45a..7f46843e7428 100644
--- a/tools/testing/selftests/kvm/Makefile.kvm
+++ b/tools/testing/selftests/kvm/Makefile.kvm
@@ -77,6 +77,12 @@ TEST_GEN_PROGS_x86 += x86/evmcs_smm_controls_test
 TEST_GEN_PROGS_x86 += x86/exit_on_emulation_failure_test
 TEST_GEN_PROGS_x86 += x86/fastops_test
 TEST_GEN_PROGS_x86 += x86/fix_hypercall_test
+TEST_GEN_PROGS_x86 += x86/gmem_provider_test
+TEST_GEN_PROGS_x86 += x86/gmem_provider_hugepage_test
+TEST_GEN_PROGS_x86 += x86/gmem_provider_revoke_test
+TEST_GEN_PROGS_x86 += x86/gmem_provider_iommufd_test
+TEST_GEN_PROGS_x86 += x86/gmem_provider_vfio_test
+TEST_GEN_PROGS_x86 += x86/gmem_provider_nvme_dma_test
 TEST_GEN_PROGS_x86 += x86/hwcr_msr_test
 TEST_GEN_PROGS_x86 += x86/hyperv_clock
 TEST_GEN_PROGS_x86 += x86/hyperv_cpuid
@@ -161,9 +167,6 @@ TEST_GEN_PROGS_x86 += rseq_test
 TEST_GEN_PROGS_x86 += steal_time
 TEST_GEN_PROGS_x86 += system_counter_offset_test
 TEST_GEN_PROGS_x86 += pre_fault_memory_test
-TEST_GEN_PROGS_x86 += x86/gmem_provider_test
-TEST_GEN_PROGS_x86 += x86/gmem_provider_hugepage_test
-TEST_GEN_PROGS_x86 += x86/gmem_provider_revoke_test
 
 # Compiled outputs used by test targets
 TEST_GEN_PROGS_EXTENDED_x86 += x86/nx_huge_pages_test
diff --git a/tools/testing/selftests/kvm/x86/gmem_provider_iommufd_test.c b/tools/testing/selftests/kvm/x86/gmem_provider_iommufd_test.c
new file mode 100644
index 000000000000..0b2fc097b5a9
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/gmem_provider_iommufd_test.c
@@ -0,0 +1,150 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * gmem_provider_iommufd_test - map the SAME provider-backed region into KVM
+ * (as a guest_memfd memslot) AND into an iommufd IOAS (via a dma-buf exported
+ * from the provider fd).  Prove both APIs accept the mapping and that a guest
+ * write is visible via the host mmap.
+ *
+ * The iommufd side exercises exactly the provider->iommufd path we just wired:
+ * GET_DMABUF on the provider fd -> IOMMU_IOAS_MAP_FILE, which walks
+ * iopt_map_dmabuf -> sym_..._iommufd_map -> gmem_provider_dma_buf_iommufd_map.
+ *
+ * The test opens the provider with GMEM_PROVIDER_FLAG_MMAP_CAPABLE at SETUP time; requires iommufd
+ * available at /dev/iommu.  Actual IOMMU page-table programming happens once
+ * a device is attached; here we validate the plumbing (map succeeds; unmap
+ * returns the same byte count) without needing a passthrough device.
+ */
+#include <fcntl.h>
+#include <errno.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <string.h>
+#include <unistd.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+
+#include <linux/iommufd.h>
+
+#include "test_util.h"
+#include "kvm_util.h"
+#include "processor.h"
+
+/* Mirrors samples/kvm/gmem_provider.h */
+struct gmem_provider_setup {
+	__s32 kvm_fd;
+	__u32 flags;
+	__u64 size;
+};
+#define GMEM_PROVIDER_SETUP		_IOW('G', 1, struct gmem_provider_setup)
+#define GMEM_PROVIDER_FLAG_MMAP_CAPABLE	(1u << 0)
+#define GMEM_PROVIDER_GET_DMABUF	_IO('G', 3)
+
+#define DATA_SLOT	10
+#define DATA_GPA	(1ULL << 32)
+#define DATA_SIZE	0x200000ULL		/* 2 MiB */
+#define IOVA_BASE	(1ULL << 34)		/* arbitrary */
+#define MAGIC		0xa5a5a5a5a5a5a5a5ULL
+
+static void guest_code(void)
+{
+	*(volatile uint64_t *)DATA_GPA = MAGIC;
+	GUEST_DONE();
+}
+
+int main(void)
+{
+	struct vm_shape shape = {
+		.mode = VM_MODE_DEFAULT,
+		.type = KVM_X86_SW_PROTECTED_VM,
+	};
+	struct gmem_provider_setup setup = { .flags = GMEM_PROVIDER_FLAG_MMAP_CAPABLE };
+	struct iommu_ioas_alloc alloc = {};
+	struct iommu_ioas_map_file map = {};
+	struct kvm_vcpu *vcpu;
+	struct kvm_vm *vm;
+	struct ucall uc;
+	int gmem_ctl, gmem_fd, dmabuf_fd, iommufd, r;
+	void *hva;
+
+	TEST_REQUIRE(kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SW_PROTECTED_VM));
+
+	gmem_ctl = open("/dev/gmem_provider", O_RDWR);
+	__TEST_REQUIRE(gmem_ctl >= 0,
+		       "gmem_provider not loaded (/dev/gmem_provider absent)");
+	iommufd = open("/dev/iommu", O_RDWR);
+	__TEST_REQUIRE(iommufd >= 0, "iommufd unavailable (/dev/iommu absent)");
+
+	/*
+	 * 1) Provider fd bound to KVM VM.
+	 */
+	vm = vm_create_shape_with_one_vcpu(shape, &vcpu, guest_code);
+	setup.kvm_fd = vm->fd;
+	setup.size = DATA_SIZE;
+	gmem_fd = ioctl(gmem_ctl, GMEM_PROVIDER_SETUP, &setup);
+	TEST_ASSERT(gmem_fd >= 0, "GMEM_PROVIDER_SETUP failed errno=%d", errno);
+
+	/*
+	 * 2) KVM side: host mmap + guest_memfd memslot (gmem-only when mmap-capable).
+	 */
+	hva = mmap(NULL, DATA_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, gmem_fd, 0);
+	TEST_ASSERT(hva != MAP_FAILED,
+		    "provider mmap failed errno=%d", errno);
+	r = __vm_set_user_memory_region2(vm, DATA_SLOT, KVM_MEM_GUEST_MEMFD,
+					 DATA_GPA, DATA_SIZE, hva, gmem_fd, 0);
+	TEST_ASSERT(!r, "KVM_SET_USER_MEMORY_REGION2 failed r=%d errno=%d", r, errno);
+	virt_map(vm, DATA_GPA, DATA_GPA, 1);
+
+	/*
+	 * 3) iommufd side: allocate IOAS, get a dma-buf from the SAME provider fd,
+	 *    and map it into the IOAS via IOMMU_IOAS_MAP_FILE.  This drives
+	 *    iopt_map_dmabuf -> gmem_provider_dma_buf_iommufd_map.
+	 */
+	alloc.size = sizeof(alloc);
+	r = ioctl(iommufd, IOMMU_IOAS_ALLOC, &alloc);
+	TEST_ASSERT(!r, "IOMMU_IOAS_ALLOC failed errno=%d", errno);
+	pr_info("iommufd: allocated ioas id=%u\n", alloc.out_ioas_id);
+
+	dmabuf_fd = ioctl(gmem_fd, GMEM_PROVIDER_GET_DMABUF);
+	TEST_ASSERT(dmabuf_fd >= 0, "GMEM_PROVIDER_GET_DMABUF failed errno=%d", errno);
+	pr_info("provider: exported dma-buf fd=%d\n", dmabuf_fd);
+
+	map.size = sizeof(map);
+	map.flags = IOMMU_IOAS_MAP_FIXED_IOVA |
+		    IOMMU_IOAS_MAP_READABLE | IOMMU_IOAS_MAP_WRITEABLE;
+	map.ioas_id = alloc.out_ioas_id;
+	map.fd = dmabuf_fd;
+	map.start = 0;
+	map.length = DATA_SIZE;
+	map.iova = IOVA_BASE;
+	r = ioctl(iommufd, IOMMU_IOAS_MAP_FILE, &map);
+	TEST_ASSERT(!r,
+		    "IOMMU_IOAS_MAP_FILE(dma-buf) failed r=%d errno=%d\n"
+		    "	 (gmem_provider_dma_buf_iommufd_map path)",
+		    r, errno);
+	pr_info("iommufd: mapped provider dma-buf @ IOVA 0x%llx (0x%llx bytes)\n",
+		(unsigned long long)map.iova, (unsigned long long)map.length);
+
+	/*
+	 * 4) Run the guest.  It writes MAGIC via the KVM/NPT path; we then read
+	 *    via the host mmap to confirm the dual-mapping resolves to the same
+	 *    physical backing.
+	 */
+	vcpu_run(vcpu);
+	TEST_ASSERT(get_ucall(vcpu, &uc) == UCALL_DONE,
+		    "guest exit: %s", exit_reason_str(vcpu->run->exit_reason));
+
+	TEST_ASSERT(*(volatile uint64_t *)hva == MAGIC,
+		    "host mmap read 0x%llx, want MAGIC (guest+host+iommufd disagree)",
+		    (unsigned long long)*(volatile uint64_t *)hva);
+	pr_info("dual-map ok: guest wrote MAGIC, host mmap sees 0x%llx, "
+		"iommufd IOAS holds the same backing\n",
+		(unsigned long long)*(volatile uint64_t *)hva);
+
+	kvm_vm_free(vm);
+	close(dmabuf_fd);
+	close(iommufd);
+	munmap(hva, DATA_SIZE);
+	close(gmem_fd);
+	close(gmem_ctl);
+	return 0;
+}
diff --git a/tools/testing/selftests/kvm/x86/gmem_provider_nvme_dma_test.c b/tools/testing/selftests/kvm/x86/gmem_provider_nvme_dma_test.c
new file mode 100644
index 000000000000..8c8a4b2e8ac9
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/gmem_provider_nvme_dma_test.c
@@ -0,0 +1,365 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * gmem_provider_nvme_dma_test - real DMA into provider memory via a
+ * passthrough NVMe controller.
+ *
+ * Sends an Identify Controller (opc=0x06, CNS=1) admin command to a
+ * vfio-pci-bound NVMe controller, with PRP1 pointing at an IOVA that maps
+ * to the gmem provider's backing.  Success = the provider region contains
+ * the 4 KiB Identify Controller data structure (verified via VID field
+ * matching the PCI vendor read from config space).
+ *
+ * Requires the gmem_provider module loaded, iommufd available, and an NVMe
+ * controller bound to vfio-pci at /dev/vfio/devices/vfio0 (overridable via
+ * GMEM_VFIO_CDEV).  DESTROYS ANY IN-FLIGHT STATE on that controller.
+ */
+#include <fcntl.h>
+#include <errno.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+
+#include <linux/iommufd.h>
+#include <linux/vfio.h>
+#include <linux/kvm.h>
+
+#include "test_util.h"
+
+struct gmem_provider_setup {
+	__s32 kvm_fd;
+	__u32 flags;
+	__u64 size;
+};
+#define GMEM_PROVIDER_SETUP		_IOW('G', 1, struct gmem_provider_setup)
+#define GMEM_PROVIDER_FLAG_MMAP_CAPABLE	(1u << 0)
+#define GMEM_PROVIDER_GET_DMABUF	_IO('G', 3)
+
+struct gmem_provider_present {
+	__u64 offset;
+	__u64 len;
+	__u32 present;
+	__u32 pad;
+};
+#define GMEM_PROVIDER_SET_PRESENT	_IOW('G', 2, struct gmem_provider_present)
+
+#define PROVIDER_SIZE	0x200000ULL		/* 2 MiB provider region */
+#define IOVA_DST	0x40000000ULL		/* target for Identify Controller data */
+#define IOVA_SQ		(IOVA_DST + PROVIDER_SIZE)
+#define IOVA_CQ		(IOVA_SQ + 0x1000)
+
+/* NVMe controller regs (BAR0). */
+#define NVME_REG_CAP	0x00
+#define NVME_REG_VS	0x08
+#define NVME_REG_CC	0x14
+#define NVME_REG_CSTS	0x1C
+#define NVME_REG_AQA	0x24
+#define NVME_REG_ASQ	0x28
+#define NVME_REG_ACQ	0x30
+#define NVME_SQ0TDBL	0x1000
+
+#define NVME_CC_EN	(1U << 0)
+#define NVME_CSTS_RDY	(1U << 0)
+
+static uint32_t rd32(void *b, uint32_t o) { return *(volatile uint32_t *)((char *)b + o); }
+static uint64_t rd64(void *b, uint32_t o) { return *(volatile uint64_t *)((char *)b + o); }
+static void     wr32(void *b, uint32_t o, uint32_t v) { *(volatile uint32_t *)((char *)b + o) = v; }
+static void     wr64(void *b, uint32_t o, uint64_t v) { *(volatile uint64_t *)((char *)b + o) = v; }
+
+static int ioas_map_user(int iommufd_fd, uint32_t ioas_id,
+			 void *va, uint64_t iova, uint64_t len)
+{
+	struct iommu_ioas_map m = {
+		.size = sizeof(m),
+		.flags = IOMMU_IOAS_MAP_FIXED_IOVA |
+			 IOMMU_IOAS_MAP_READABLE |
+			 IOMMU_IOAS_MAP_WRITEABLE,
+		.ioas_id = ioas_id,
+		.user_va = (uintptr_t)va,
+		.length = len,
+		.iova = iova,
+	};
+	return ioctl(iommufd_fd, IOMMU_IOAS_MAP, &m);
+}
+
+int main(void)
+{
+	struct gmem_provider_setup setup = { .flags = GMEM_PROVIDER_FLAG_MMAP_CAPABLE };
+	struct iommu_ioas_alloc alloc = {};
+	struct iommu_ioas_map_file mapf = {};
+	struct vfio_device_bind_iommufd bind = {};
+	struct vfio_device_attach_iommufd_pt att = {};
+	struct vfio_region_info reg = {};
+	const char *cdev_path;
+	int gmem_ctl, gmem_fd, dmabuf_fd, iommufd_fd, vfio_fd;
+	void *provider_hva, *sq_buf, *cq_buf, *bar;
+	uint16_t pci_cmd;
+	uint64_t cap, cfg_off;
+	uint32_t doorbell_stride, aqa, cc, csts;
+	int i;
+
+	gmem_ctl = open("/dev/gmem_provider", O_RDWR);
+	__TEST_REQUIRE(gmem_ctl >= 0, "gmem_provider module not loaded");
+	iommufd_fd = open("/dev/iommu", O_RDWR);
+	__TEST_REQUIRE(iommufd_fd >= 0, "iommufd unavailable");
+	cdev_path = getenv("GMEM_VFIO_CDEV");
+	if (!cdev_path)
+		cdev_path = "/dev/vfio/devices/vfio0";
+	vfio_fd = open(cdev_path, O_RDWR);
+	__TEST_REQUIRE(vfio_fd >= 0, "vfio cdev absent: %s", cdev_path);
+
+	/* Provider fd (bare KVM VM to satisfy SETUP). */
+	{
+		int kvm = open("/dev/kvm", O_RDWR);
+		int vm;
+		TEST_ASSERT(kvm >= 0, "open /dev/kvm errno=%d", errno);
+		vm = ioctl(kvm, KVM_CREATE_VM, 0);
+		TEST_ASSERT(vm >= 0, "KVM_CREATE_VM errno=%d", errno);
+		setup.kvm_fd = vm;
+		close(kvm);
+	}
+	setup.size = PROVIDER_SIZE;
+	gmem_fd = ioctl(gmem_ctl, GMEM_PROVIDER_SETUP, &setup);
+	TEST_ASSERT(gmem_fd >= 0, "SETUP errno=%d", errno);
+
+	provider_hva = mmap(NULL, PROVIDER_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED,
+			    gmem_fd, 0);
+	TEST_ASSERT(provider_hva != MAP_FAILED, "provider mmap errno=%d", errno);
+	memset(provider_hva, 0xcc, 0x1000);	/* poison first page */
+
+	/* IOAS + provider dma-buf. */
+	alloc.size = sizeof(alloc);
+	TEST_ASSERT(!ioctl(iommufd_fd, IOMMU_IOAS_ALLOC, &alloc), "IOAS_ALLOC");
+	dmabuf_fd = ioctl(gmem_fd, GMEM_PROVIDER_GET_DMABUF);
+	TEST_ASSERT(dmabuf_fd >= 0, "GET_DMABUF");
+	mapf.size = sizeof(mapf);
+	mapf.flags = IOMMU_IOAS_MAP_FIXED_IOVA | IOMMU_IOAS_MAP_READABLE |
+		     IOMMU_IOAS_MAP_WRITEABLE;
+	mapf.ioas_id = alloc.out_ioas_id;
+	mapf.fd = dmabuf_fd;
+	mapf.length = PROVIDER_SIZE;
+	mapf.iova = IOVA_DST;
+	TEST_ASSERT(!ioctl(iommufd_fd, IOMMU_IOAS_MAP_FILE, &mapf), "IOAS_MAP_FILE");
+
+	/* Admin SQ/CQ backing pages (userspace anon, IOMMU-mapped). */
+	sq_buf = mmap(NULL, 0x1000, PROT_READ | PROT_WRITE,
+		      MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+	TEST_ASSERT(sq_buf != MAP_FAILED, "sq mmap");
+	memset(sq_buf, 0, 0x1000);
+	cq_buf = mmap(NULL, 0x1000, PROT_READ | PROT_WRITE,
+		      MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+	TEST_ASSERT(cq_buf != MAP_FAILED, "cq mmap");
+	memset(cq_buf, 0, 0x1000);
+	TEST_ASSERT(!ioas_map_user(iommufd_fd, alloc.out_ioas_id, sq_buf, IOVA_SQ, 0x1000),
+		    "IOAS_MAP(sq) errno=%d", errno);
+	TEST_ASSERT(!ioas_map_user(iommufd_fd, alloc.out_ioas_id, cq_buf, IOVA_CQ, 0x1000),
+		    "IOAS_MAP(cq) errno=%d", errno);
+
+	/* Bind + attach. */
+	bind.argsz = sizeof(bind);
+	bind.iommufd = iommufd_fd;
+	TEST_ASSERT(!ioctl(vfio_fd, VFIO_DEVICE_BIND_IOMMUFD, &bind), "BIND");
+	att.argsz = sizeof(att);
+	att.pt_id = alloc.out_ioas_id;
+	TEST_ASSERT(!ioctl(vfio_fd, VFIO_DEVICE_ATTACH_IOMMUFD_PT, &att), "ATTACH");
+	pr_info("vfio device attached, pt_id=%u\n", att.pt_id);
+
+	/* mmap BAR0. */
+	reg.argsz = sizeof(reg);
+	reg.index = VFIO_PCI_BAR0_REGION_INDEX;
+	TEST_ASSERT(!ioctl(vfio_fd, VFIO_DEVICE_GET_REGION_INFO, &reg), "GET_REGION_INFO");
+	__TEST_REQUIRE(reg.flags & VFIO_REGION_INFO_FLAG_MMAP, "BAR0 not mmap-capable");
+	bar = mmap(NULL, reg.size, PROT_READ | PROT_WRITE, MAP_SHARED, vfio_fd, reg.offset);
+	TEST_ASSERT(bar != MAP_FAILED, "BAR0 mmap");
+	pr_info("BAR0 mapped size=0x%llx\n", (unsigned long long)reg.size);
+
+	/* Enable Bus Master. */
+	reg.argsz = sizeof(reg);
+	reg.index = VFIO_PCI_CONFIG_REGION_INDEX;
+	TEST_ASSERT(!ioctl(vfio_fd, VFIO_DEVICE_GET_REGION_INFO, &reg), "CFG region info");
+	cfg_off = reg.offset;
+	TEST_ASSERT(pread(vfio_fd, &pci_cmd, 2, cfg_off + 0x04) == 2, "cfg CMD read");
+	pci_cmd |= 0x0006;
+	TEST_ASSERT(pwrite(vfio_fd, &pci_cmd, 2, cfg_off + 0x04) == 2, "cfg CMD write");
+
+	cap = rd64(bar, NVME_REG_CAP);
+	doorbell_stride = 4U << ((cap >> 32) & 0xf);
+	pr_info("NVMe CAP=0x%016llx VS=0x%08x MQES=%u DSTRD=%u\n",
+		(unsigned long long)cap, rd32(bar, NVME_REG_VS),
+		(uint32_t)((cap & 0xffff) + 1), doorbell_stride);
+
+	/* Disable + wait !RDY. */
+	wr32(bar, NVME_REG_CC, 0);
+	for (i = 0; i < 5000 && (rd32(bar, NVME_REG_CSTS) & NVME_CSTS_RDY); i++)
+		usleep(1000);
+
+	/* Admin queues: 64 entries each. */
+	wr64(bar, NVME_REG_ASQ, IOVA_SQ);
+	wr64(bar, NVME_REG_ACQ, IOVA_CQ);
+	aqa = (63U << 16) | 63U;			/* ACQS-1 | ASQS-1 */
+	wr32(bar, NVME_REG_AQA, aqa);
+	cc = (6U << 16) | (4U << 20) | NVME_CC_EN;	/* IOSQES=6 IOCQES=4 EN */
+	wr32(bar, NVME_REG_CC, cc);
+
+	for (i = 0; i < 10000; i++) {
+		csts = rd32(bar, NVME_REG_CSTS);
+		if (csts & NVME_CSTS_RDY)
+			break;
+		usleep(1000);
+		if (csts & 0x02) {	/* CFS */
+			TEST_FAIL("NVMe controller fatal status CSTS=0x%x", csts);
+		}
+	}
+	TEST_ASSERT(csts & NVME_CSTS_RDY, "controller never reached RDY, CSTS=0x%x", csts);
+	pr_info("NVMe ready. CSTS=0x%x\n", csts);
+
+	/* Identify Controller SQE at slot 0. */
+	{
+		uint32_t *s = (uint32_t *)sq_buf;
+		memset(s, 0, 64);
+		s[0]  = 0x00010006;	/* CID=1 << 16 | OPC=Identify */
+		s[1]  = 0;		/* NSID */
+		/* PRP1 (offset 24 in SQE = dword 6) */
+		s[6]  = (uint32_t)IOVA_DST;
+		s[7]  = (uint32_t)(IOVA_DST >> 32);
+		s[8]  = 0;
+		s[9]  = 0;		/* PRP2 */
+		s[10] = 0x01;		/* CDW10: CNS = Identify Controller */
+	}
+
+	/* Ring SQ0 tail doorbell. */
+	wr32(bar, NVME_SQ0TDBL, 1);
+	pr_info("Identify Controller submitted (SQ0 tail=1)\n");
+
+	/* Poll CQ0 slot 0 for a valid completion (phase tag = 1 on first pass). */
+	{
+		volatile uint32_t *c = (volatile uint32_t *)cq_buf;
+		uint32_t status;
+		for (i = 0; i < 5000; i++) {
+			if (c[3] & 0x10000)		/* P bit */
+				break;
+			usleep(1000);
+		}
+		TEST_ASSERT(c[3] & 0x10000, "completion phase bit never set");
+		status = (c[3] >> 17) & 0x7fff;
+		pr_info("CQE: DW0=0x%08x DW1=0x%08x DW2=0x%08x DW3=0x%08x  status=0x%04x\n",
+			c[0], c[1], c[2], c[3], status);
+		TEST_ASSERT(status == 0, "Identify Controller status non-zero: 0x%x", status);
+		fprintf(stderr, "MARKER: about to write CQ doorbell\n"); fflush(stderr);
+		/* CQ0 head doorbell = 1 */
+		wr32(bar, NVME_SQ0TDBL + doorbell_stride, 1);
+		fprintf(stderr, "MARKER: about to read provider\n"); fflush(stderr);
+	}
+
+	/* Provider region now holds Identify Controller data.  VID/SSVID at bytes 0..3
+	 * must equal the device's PCI vendor + subsystem vendor (both are Amazon here). */
+	{
+		uint16_t vid  = ((uint16_t *)provider_hva)[0];
+		uint16_t ssvid = ((uint16_t *)provider_hva)[1];
+		uint8_t first = ((uint8_t *)provider_hva)[0];
+		uint8_t poison_still = ((uint8_t *)provider_hva)[8];
+		char mn[41] = {};
+		memcpy(mn, (char *)provider_hva + 24, 40);
+		fprintf(stderr, "MARKER: read done; byte0=0x%02x byte8=0x%02x\n",
+			first, poison_still);
+		fflush(stderr);
+		pr_info("Identify Controller in provider: VID=0x%04x SSVID=0x%04x MN='%s'\n",
+			vid, ssvid, mn);
+		fflush(stdout);
+		TEST_ASSERT(vid == 0x1d0f,
+			    "unexpected VID 0x%04x in provider region -- Identify DMA landed elsewhere?\n"
+			    "  poison=0x%02x byte0=0x%02x (poison intact=DMA didn't land here)",
+			    vid, poison_still, first);
+		pr_info("NVMe -> provider DMA VERIFIED\n");
+	}
+
+	/*
+	 * Phase 2: revoke the target page in the provider and verify that the
+	 * next DMA to the same IOVA fails, without touching provider memory.
+	 *
+	 * SET_PRESENT(present=0) marks the range absent in the provider bitmap
+	 * and fans out dma_buf_invalidate_mappings() to every exported dma-buf.
+	 * iommufd's iopt_revoke_notify (registered as .invalidate_mappings on
+	 * the dynamic attach) tears down the IOMMU mapping for that range and
+	 * marks the pages as revoked, so a subsequent device DMA to the same
+	 * IOVA lands with no IOMMU translation and either faults at the IOMMU
+	 * or completes with an NVMe error.  Either way the provider region
+	 * must not be overwritten.
+	 */
+	{
+		struct gmem_provider_present rev = {
+			.offset = 0, .len = 0x1000, .present = 0,
+		};
+		volatile uint32_t *c = (volatile uint32_t *)cq_buf;
+		uint32_t *s = (uint32_t *)sq_buf;
+		uint32_t status;
+		int ret;
+
+		/* Re-seed the target page so we can detect *any* write to it. */
+		memset(provider_hva, 0xa5, 0x1000);
+
+		ret = ioctl(gmem_fd, GMEM_PROVIDER_SET_PRESENT, &rev);
+		TEST_ASSERT(ret == 0, "SET_PRESENT(revoke) errno=%d", errno);
+		pr_info("Revoked provider offset=0 len=0x1000; dma-buf mappings invalidated\n");
+
+		/* Issue a second Identify Controller aimed at the revoked IOVA.
+		 * Use a new CID and place the SQE in slot 1; the SQ ring will
+		 * wrap for larger runs but we only ever use slots 0 and 1.
+		 */
+		memset(&s[16], 0, 64);
+		s[16] = 0x00020006;			/* CID=2 << 16 | OPC=Identify */
+		s[16 + 6] = (uint32_t)IOVA_DST;
+		s[16 + 7] = (uint32_t)(IOVA_DST >> 32);
+		s[16 + 10] = 0x01;			/* CNS = Identify Controller */
+
+		wr32(bar, NVME_SQ0TDBL, 2);
+		pr_info("Identify Controller (revoked target) submitted (SQ0 tail=2)\n");
+
+		/* Poll for CQE at slot 1 with phase bit inverted from first pass. */
+		for (i = 0; i < 5000; i++) {
+			if ((c[4 + 3] & 0x10000))	/* CQE 1 P bit */
+				break;
+			usleep(1000);
+		}
+		TEST_ASSERT(c[4 + 3] & 0x10000, "revoke-phase CQE never landed");
+		status = (c[4 + 3] >> 17) & 0x7fff;
+		pr_info("post-revoke CQE: DW0=0x%08x DW3=0x%08x  status=0x%04x\n",
+			c[4 + 0], c[4 + 3], status);
+		wr32(bar, NVME_SQ0TDBL + doorbell_stride, 2);
+
+		/* Two acceptable outcomes here: (a) NVMe reports a non-zero
+		 * status (typical when the IOMMU fault takes the transaction
+		 * down), or (b) NVMe reports success but the target page was
+		 * not overwritten because the IOMMU rerouted / dropped the
+		 * write.  Either proves the revocation took effect on the DMA
+		 * side.  A silent success that overwrites provider memory
+		 * would be a bug in our revoke plumbing.
+		 */
+		{
+			uint8_t b0 = ((uint8_t *)provider_hva)[0];
+			pr_info("post-revoke provider byte0=0x%02x (seeded 0xa5)\n", b0);
+			TEST_ASSERT(b0 == 0xa5 || status != 0,
+				    "revoke leaked: status=0 and provider byte0=0x%02x != 0xa5",
+				    b0);
+			pr_info("Revoked-page DMA is not observable in provider memory\n");
+		}
+
+		/* Restore and confirm a fresh IOMMU_IOAS_MAP_FILE succeeds. */
+		rev.present = 1;
+		ret = ioctl(gmem_fd, GMEM_PROVIDER_SET_PRESENT, &rev);
+		TEST_ASSERT(ret == 0, "SET_PRESENT(restore) errno=%d", errno);
+		pr_info("Restore + revoke round trip complete\n");
+	}
+
+	munmap(bar, reg.size);
+	close(vfio_fd);
+	close(dmabuf_fd);
+	close(iommufd_fd);
+	munmap(provider_hva, PROVIDER_SIZE);
+	close(gmem_fd);
+	close(gmem_ctl);
+	return 0;
+}
diff --git a/tools/testing/selftests/kvm/x86/gmem_provider_vfio_test.c b/tools/testing/selftests/kvm/x86/gmem_provider_vfio_test.c
new file mode 100644
index 000000000000..97a66a295b14
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/gmem_provider_vfio_test.c
@@ -0,0 +1,134 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * gmem_provider_vfio_test - real passthrough device attached to an iommufd
+ * IOAS holding the provider dma-buf.
+ *
+ * Steps:
+ *   1. SETUP provider (KVM VM shape not required for this test).
+ *   2. iommufd IOAS + GET_DMABUF + IOMMU_IOAS_MAP_FILE - IOAS holds the
+ *      provider region.
+ *   3. Open a vfio-pci cdev (default /dev/vfio/devices/vfio0, overridable
+ *      via GMEM_VFIO_CDEV env), VFIO_DEVICE_BIND_IOMMUFD, then
+ *      VFIO_DEVICE_ATTACH_IOMMUFD_PT to the IOAS.
+ *
+ * On success the kernel has allocated (or reused) an iommu_domain for the
+ * device covering the provider region: subsequent DMA by the device to the
+ * mapped IOVAs would land in provider memory.  Actual DMA descriptor
+ * programming is device-specific and left as a follow-on.
+ *
+ * Skips if either the provider module or a bound vfio-pci cdev is absent.
+ */
+#include <fcntl.h>
+#include <errno.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <sys/ioctl.h>
+
+#include <linux/iommufd.h>
+#include <linux/vfio.h>
+#include <linux/kvm.h>
+
+#include "test_util.h"
+
+struct gmem_provider_setup {
+	__s32 kvm_fd;
+	__u32 flags;
+	__u64 size;
+};
+#define GMEM_PROVIDER_SETUP		_IOW('G', 1, struct gmem_provider_setup)
+#define GMEM_PROVIDER_FLAG_MMAP_CAPABLE	(1u << 0)
+#define GMEM_PROVIDER_GET_DMABUF	_IO('G', 3)
+
+#define DATA_SIZE	0x200000ULL
+#define IOVA_BASE	(1ULL << 34)
+
+int main(void)
+{
+	struct gmem_provider_setup setup = { .flags = GMEM_PROVIDER_FLAG_MMAP_CAPABLE };
+	struct iommu_ioas_alloc alloc = {};
+	struct iommu_ioas_map_file map = {};
+	struct vfio_device_bind_iommufd bind = {};
+	struct vfio_device_attach_iommufd_pt att = {};
+	const char *cdev_path;
+	int gmem_ctl, gmem_fd, dmabuf_fd, iommufd_fd, vfio_fd, r;
+
+	gmem_ctl = open("/dev/gmem_provider", O_RDWR);
+	__TEST_REQUIRE(gmem_ctl >= 0, "gmem_provider module not loaded");
+	iommufd_fd = open("/dev/iommu", O_RDWR);
+	__TEST_REQUIRE(iommufd_fd >= 0, "iommufd unavailable (/dev/iommu absent)");
+
+	cdev_path = getenv("GMEM_VFIO_CDEV");
+	if (!cdev_path)
+		cdev_path = "/dev/vfio/devices/vfio0";
+	vfio_fd = open(cdev_path, O_RDWR);
+	__TEST_REQUIRE(vfio_fd >= 0,
+		       "no vfio-pci cdev at %s (bind a device first)", cdev_path);
+	pr_info("using vfio cdev: %s (fd=%d)\n", cdev_path, vfio_fd);
+
+	/* Provider fd (KVM binding not needed here, but SETUP still requires a
+	 * valid kvm_fd; pass /dev/kvm's own fd -- the SETUP path just fgets it
+	 * to grab a KVM ref, and /dev/kvm's file has ->private_data==NULL, so
+	 * the "kvm not attached" check must be relaxed OR we create a VM.
+	 * For robustness, create a bare KVM VM. */
+	{
+		int kvm = open("/dev/kvm", O_RDWR);
+		int vm;
+		TEST_ASSERT(kvm >= 0, "open /dev/kvm errno=%d", errno);
+		vm = ioctl(kvm, KVM_CREATE_VM, 0);
+		TEST_ASSERT(vm >= 0, "KVM_CREATE_VM errno=%d", errno);
+		setup.kvm_fd = vm;
+		close(kvm);
+	}
+	setup.size = DATA_SIZE;
+	gmem_fd = ioctl(gmem_ctl, GMEM_PROVIDER_SETUP, &setup);
+	TEST_ASSERT(gmem_fd >= 0, "GMEM_PROVIDER_SETUP errno=%d", errno);
+
+	/* IOAS + provider dma-buf */
+	alloc.size = sizeof(alloc);
+	r = ioctl(iommufd_fd, IOMMU_IOAS_ALLOC, &alloc);
+	TEST_ASSERT(!r, "IOMMU_IOAS_ALLOC errno=%d", errno);
+
+	dmabuf_fd = ioctl(gmem_fd, GMEM_PROVIDER_GET_DMABUF);
+	TEST_ASSERT(dmabuf_fd >= 0, "GET_DMABUF errno=%d", errno);
+
+	map.size = sizeof(map);
+	map.flags = IOMMU_IOAS_MAP_FIXED_IOVA |
+		    IOMMU_IOAS_MAP_READABLE | IOMMU_IOAS_MAP_WRITEABLE;
+	map.ioas_id = alloc.out_ioas_id;
+	map.fd = dmabuf_fd;
+	map.length = DATA_SIZE;
+	map.iova = IOVA_BASE;
+	r = ioctl(iommufd_fd, IOMMU_IOAS_MAP_FILE, &map);
+	TEST_ASSERT(!r, "IOMMU_IOAS_MAP_FILE errno=%d", errno);
+	pr_info("IOAS %u holds provider dma-buf @ IOVA 0x%llx (0x%llx bytes)\n",
+		alloc.out_ioas_id,
+		(unsigned long long)map.iova, (unsigned long long)map.length);
+
+	/* Bind VFIO cdev to iommufd */
+	bind.argsz = sizeof(bind);
+	bind.iommufd = iommufd_fd;
+	r = ioctl(vfio_fd, VFIO_DEVICE_BIND_IOMMUFD, &bind);
+	TEST_ASSERT(!r, "VFIO_DEVICE_BIND_IOMMUFD errno=%d", errno);
+	pr_info("VFIO device bound: devid=%u\n", bind.out_devid);
+
+	/* Attach the device to our IOAS */
+	att.argsz = sizeof(att);
+	att.pt_id = alloc.out_ioas_id;
+	r = ioctl(vfio_fd, VFIO_DEVICE_ATTACH_IOMMUFD_PT, &att);
+	TEST_ASSERT(!r,
+		    "VFIO_DEVICE_ATTACH_IOMMUFD_PT errno=%d\n"
+		    "	 (real device -> IOAS attach failed)", errno);
+	pr_info("VFIO device attached to IOAS -> pt_id=%u (kernel-selected)\n",
+		att.pt_id);
+	pr_info("real passthrough device now has IOMMU domain covering provider region\n");
+
+	close(vfio_fd);
+	close(dmabuf_fd);
+	close(iommufd_fd);
+	close(gmem_fd);
+	close(gmem_ctl);
+	return 0;
+}
-- 
2.54.0

[RFC PATCH 11/11] samples/kvm, selftests/kvm: Allow the gmem_provider NVMe DMA test on arm64

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

The sample provider and the NVMe DMA selftest have no x86 specifics.
Drop the X86_64 gate on the sample's Kconfig (arm64 also has
KVM_GUEST_MEMFD; the SEV RMP reset is already guarded by
CONFIG_AMD_MEM_ENCRYPT) and move the NVMe DMA selftest from
TEST_GEN_PROGS_x86 to TEST_GEN_PROGS_COMMON so it also builds on
architectures with an IOMMU (arm64 SMMU-v3, ...).

The test itself does not need to change: it uses KVM_CREATE_VM(0) as
a client for the sample provider's SETUP ioctl and then exercises
the iommufd + dma-buf + provider revoke path with direct NVMe MMIO,
none of which is x86-specific.  On arm64 it validates the same flow
through the ARM SMMU.

Signed-off-by: David Woodhouse (Kiro) <dwmw@amazon.co.uk>
---
 samples/Kconfig                                                 | 2 +-
 tools/testing/selftests/kvm/Makefile.kvm                        | 2 +-
 .../selftests/kvm/{x86 => }/gmem_provider_nvme_dma_test.c       | 0
 3 files changed, 2 insertions(+), 2 deletions(-)
 rename tools/testing/selftests/kvm/{x86 => }/gmem_provider_nvme_dma_test.c (100%)

diff --git a/samples/Kconfig b/samples/Kconfig
index 3482659638b5..d26a03dea072 100644
--- a/samples/Kconfig
+++ b/samples/Kconfig
@@ -328,7 +328,7 @@ source "samples/damon/Kconfig"
 
 config SAMPLE_KVM_GMEM_PROVIDER
 	tristate "Build sample guest_memfd provider -- loadable module only"
-	depends on KVM_GUEST_MEMFD && X86_64 && m
+	depends on KVM_GUEST_MEMFD && (X86_64 || ARM64) && m
 	help
 	  This builds a sample external guest_memfd provider module with two
 	  backing modes.  Loaded with addr=/len= it backs guest memory with a
diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm
index 7f46843e7428..12004a487c32 100644
--- a/tools/testing/selftests/kvm/Makefile.kvm
+++ b/tools/testing/selftests/kvm/Makefile.kvm
@@ -58,6 +58,7 @@ TEST_PROGS_x86 += x86/nx_huge_pages_test.sh
 # Compiled test targets valid on all architectures with libkvm support
 TEST_GEN_PROGS_COMMON = demand_paging_test
 TEST_GEN_PROGS_COMMON += dirty_log_test
+TEST_GEN_PROGS_COMMON += gmem_provider_nvme_dma_test
 TEST_GEN_PROGS_COMMON += guest_print_test
 TEST_GEN_PROGS_COMMON += irqfd_test
 TEST_GEN_PROGS_COMMON += kvm_binary_stats_test
@@ -82,7 +83,6 @@ TEST_GEN_PROGS_x86 += x86/gmem_provider_hugepage_test
 TEST_GEN_PROGS_x86 += x86/gmem_provider_revoke_test
 TEST_GEN_PROGS_x86 += x86/gmem_provider_iommufd_test
 TEST_GEN_PROGS_x86 += x86/gmem_provider_vfio_test
-TEST_GEN_PROGS_x86 += x86/gmem_provider_nvme_dma_test
 TEST_GEN_PROGS_x86 += x86/hwcr_msr_test
 TEST_GEN_PROGS_x86 += x86/hyperv_clock
 TEST_GEN_PROGS_x86 += x86/hyperv_cpuid
diff --git a/tools/testing/selftests/kvm/x86/gmem_provider_nvme_dma_test.c b/tools/testing/selftests/kvm/gmem_provider_nvme_dma_test.c
similarity index 100%
rename from tools/testing/selftests/kvm/x86/gmem_provider_nvme_dma_test.c
rename to tools/testing/selftests/kvm/gmem_provider_nvme_dma_test.c
-- 
2.54.0

Re: [RFC PATCH 04/11] KVM: guest_memfd: Introduce guest memory ops and route native gmem through them

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
- [Critical] Dropped page reference in `kvm_gmem_populate_one` causes a TOCTOU Use-After-Free on physical pages.
- [High] Use-After-Free (UAF) in `kvm_gmem_unbind` due to lockless dereference of unrefcounted `slot->gmem.file`.
- [High] Missing module reference count in `kvm_gmem_ops` allows foreign backend unload, causing kernel panic.
- [High] `kvm_gmem_bind` API contract fails to initialize `slot->gmem.file`, breaking KVM core state tracking.
--

--- Patch [4]: [RFC PATCH 04/11] KVM: guest_memfd: Introduce guest memory ops and route native gmem through them ---
commit 4ae3212093e20a291a948708162e27b1bb7009da
Author: David Woodhouse <dwmw@amazon.co.uk>

KVM: guest_memfd: Introduce guest memory ops and route native gmem through them

This commit introduces a kvm_gmem_ops vtable to allow other subsystems to
supply guest memory backing. It routes the built-in guest_memfd operations
through this vtable and refactors the generic bind, unbind, and populate
flows.

> diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
> index ab8cfaec82d31..b33d2f4750426 100644
> --- a/include/linux/kvm_host.h
> +++ b/include/linux/kvm_host.h
[ ... ]
> @@ -621,6 +621,88 @@ static inline bool kvm_slot_has_gmem(const struct kvm_memory_slot *slot)
>  	return slot && (slot->flags & KVM_MEM_GUEST_MEMFD);
>  }
>  
> +#ifdef CONFIG_KVM_GUEST_MEMFD
> +/*
> + * struct kvm_gmem_ops - vtable a guest_memfd backing implementation exposes.
> + *
> + * Every file that can back KVM guest memory -- KVM's own guest_memfd (see
> + * kvm_gmem_create()) and any other subsystem opting in -- is created with
> + * the shared kvm_gmem_fops (see is_kvm_gmem_file()).  file->private_data on
> + * such a file points at a struct kvm_gmem_backing whose ->ops field is the
> + * per-instance vtable below.  KVM never looks any further; the rest of the
> + * per-file state (allocator, refcount, private xarray, ...) is up to the
> + * implementation and is reached via container_of() from the backing.
> + *
> + * The memory-side callbacks (bind, unbind, get_pfn, populate, invalidate)
> + * are the reason the interface exists.  The file-level callbacks (release,
> + * mmap, fallocate, ...) are dispatched by kvm_gmem_fops so that every
> + * implementation gets a private hook -- see the dispatchers in
> + * virt/kvm/guest_memfd.c.
> + *
> + * get_pfn() contract:
> + *  - *pfn is the host PFN backing @gfn.  It need not have a struct page.
> + *  - *max_order is the largest order KVM may use to map starting at @gfn.
> + *    The implementation MUST clamp it to the physical contiguity and the
> + *    HPA/GPA alignment of the backing, so KVM never builds a huge SPTE that
> + *    spans discontiguous physical memory.
> + *  - If *page is non-NULL on return the caller owns that page reference and
> + *    is responsible for put_page() after use.  If *page is left NULL the PFN
> + *    is treated as non-refcounted, and its lifetime is owned by the
> + *    implementation across bind()/unbind().
> + *
> + * Memory intended to back guest RAM MUST be reported as E820_TYPE_RAM by the
> + * host so KVM maps it write-back (and applies the memory-encryption bit on
> + * encrypted hosts) rather than treating it as MMIO.
> + */
> +struct kvm_gmem_ops {
> +	/* Memory operations. */

[Severity: High]
Is there a missing module reference count in struct kvm_gmem_ops? 

The operations table lacks an owner field (like struct module *owner) to pin
the backend module.

Since the shared kvm_gmem_fops routes file operations to a foreign backend's
vtable, could a user open the guest_memfd, unload the backend module, and
then close the file? The release dispatcher would then call into the unloaded
module's release hook, potentially causing a kernel panic.

> +	int (*bind)(struct file *file, struct kvm *kvm,
> +		    struct kvm_memory_slot *slot, loff_t offset);
[ ... ]
> diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
> index db57c5766ab60..2b277468a12f0 100644
> --- a/virt/kvm/guest_memfd.c
> +++ b/virt/kvm/guest_memfd.c
[ ... ]
> @@ -692,15 +782,51 @@ int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot,
[ ... ]
> +int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot,
> +		  unsigned int fd, uoff_t offset)
> +{
> +	const struct kvm_gmem_ops *ops;
> +	struct file *file;
> +	int r;
> +
> +	BUILD_BUG_ON(sizeof(gpa_t) != sizeof(offset));
> +	BUILD_BUG_ON(sizeof(gfn_t) != sizeof(slot->gmem.pgoff));
> +
> +	file = fget(fd);
> +	if (!file)
> +		return -EBADF;
> +
> +	ops = kvm_gmem_get_ops(file);
> +	if (!ops || !ops->bind || !ops->get_pfn) {
> +		r = -EINVAL;
> +		goto out;
> +	}
> +
> +	if (!PAGE_ALIGNED(offset)) {
> +		r = -EINVAL;
> +		goto out;
> +	}
>  
>  	/*
> -	 * Drop the reference to the file, even on success.  The file pins KVM,
> -	 * not the other way 'round.  Active bindings are invalidated if the
> -	 * file is closed before memslots are destroyed.
> +	 * On success .bind() associates the fd with @kvm and (if the
> +	 * implementation needs it) records the reference that makes the fd
> +	 * pin the VM.  A .bind() that returns success is trusted to have
> +	 * done so; there is no separate ownership check here.
> +	 */
> +	r = ops->bind(file, kvm, slot, offset);
> +out:
> +	/*
> +	 * Drop the reference to the file even on success.  The file pins
> +	 * KVM, not the other way 'round: it is the implementation's job to
> +	 * invalidate its bindings if the fd is closed before the memslots
> +	 * are destroyed.
>  	 */
> -	r = 0;
> -err:
>  	fput(file);
>  	return r;
>  }

[Severity: High]
Does this function miss initializing slot->gmem.file on a successful bind?

KVM core code relies heavily on slot->gmem.file being populated to later call
unbind. If a foreign backend succeeds in ops->bind but doesn't intentionally
mutate KVM's internal slot->gmem.file state, KVM might never call unbind
when the memslot is subsequently freed.

Could this leak backend resources or leave dangling memslot pointers in the
backend?

> @@ -741,15 +861,41 @@ void kvm_gmem_unbind(struct kvm_memory_slot *slot)
[ ... ]
> +void kvm_gmem_unbind(struct kvm *kvm, struct kvm_memory_slot *slot)
> +{
> +	const struct kvm_gmem_ops *ops;
> +	struct file *file;
> +
> +	/*
> +	 * Nothing to do if the underlying file was _already_ closed, as
> +	 * kvm_gmem_release() invalidates and nullifies all bindings.
> +	 */
> +	file = READ_ONCE(slot->gmem.file);
> +	if (!file)
> +		return;
> +
> +	ops = kvm_gmem_get_ops(file);

[Severity: High]
Is it safe to dereference the file locklessly here?

The generic code previously used get_file_active() to safely obtain a
reference. Because kvm_gmem_bind() intentionally drops its reference to the
file on success (via fput(file)), userspace can close the guest_memfd
concurrently while the memslot is being destroyed.

Could this result in a use-after-free when kvm_gmem_get_ops() attempts to
dereference the unrefcounted file pointer?

> +	if (ops && ops->unbind)
> +		ops->unbind(file, kvm, slot);
[ ... ]
> @@ -860,11 +1020,50 @@ static long __kvm_gmem_populate(struct kvm *kvm, struct kvm_memory_slot *slot,
>  	return ret;
>  }
>  
> +static int kvm_gmem_populate_one(const struct kvm_gmem_ops *ops,
> +				      struct file *file, struct kvm *kvm,
> +				      struct kvm_memory_slot *slot, gfn_t gfn,
> +				      struct page *src_page,
> +				      kvm_gmem_populate_cb post_populate,
> +				      void *opaque)
> +{
> +	kvm_pfn_t pfn;
> +	int ret;
> +
> +	/*
> +	 * Resolve the PFN via the ops.  A populate() op may also copy
> +	 * @src_page into the backing before the arch post_populate() step
> +	 * (e.g. SNP LAUNCH_UPDATE); otherwise fall back to get_pfn().
> +	 */
> +	if (ops->populate)
> +		ret = ops->populate(file, kvm, slot, gfn, &pfn,
> +				    src_page, 0);
> +	else {
> +		struct page *ignored_page = NULL;
> +
> +		ret = ops->get_pfn(file, kvm, slot, gfn, &pfn,
> +				   &ignored_page, NULL);
> +		/*
> +		 * Populate does not consume a refcount on the destination page.
> +		 * If get_pfn() handed a refcount back through @page, drop it.
> +		 * Implementations whose PFNs aren't refcounted per-call leave
> +		 * @page NULL and this is a no-op.
> +		 */
> +		if (!ret && ignored_page)
> +			put_page(ignored_page);
> +	}
> +	if (ret)
> +		return ret;
> +
> +	return post_populate(kvm, gfn, pfn, src_page, opaque);
> +}

[Severity: Critical]
Does dropping the page reference before calling post_populate() introduce a
time-of-check to time-of-use race?

If the backend provides a physical page via get_pfn() and we drop the pin
with put_page(ignored_page) early, the page could be concurrently freed and
reallocated to another process.

Could this cause the hardware post_populate callback (such as SNP RMPUPDATE)
to operate on a physical page that now belongs to an innocent host process,
corrupting host memory?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/f09647f427565c1789a6979470045aab78661563.1784194290.git.dwmw@amazon.co.uk?part=4

Re: [RFC PATCH 03/11] KVM: SEV: Remove struct page dependency from SNP gmem paths

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] An unprivileged userspace process can trigger a host kernel warning in sev_clflush_pfn(), violating KVM subsystem rules.
--

--- Patch [3]: [RFC PATCH 03/11] KVM: SEV: Remove struct page dependency from SNP gmem paths ---
commit 23550409886d9ce92ccb0c443cb1bc2d231b31b3
Author: Connor Williamson <connordw@amazon.com>

KVM: SEV: Remove struct page dependency from SNP gmem paths

This commit updates KVM SEV-SNP paths to support memory without a struct page
by falling back to memremap() instead of relying on kmap_local_page(). It
relaxes RMP helpers and updates cache flush paths to handle page-less PFNs.

> diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
> index 4272293478766..2be9e3b80d85e 100644
> --- a/arch/x86/kvm/svm/sev.c
> +++ b/arch/x86/kvm/svm/sev.c

[ ... ]

> @@ -5117,6 +5155,28 @@ int sev_gmem_prepare(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order)
>  	return 0;
>  }
>  
> +/*
> + * Flush the CPU caches for a guest_memfd PFN.  A PFN from an external
> + * guest_memfd provider may have no direct map alias, so fall back to a
> + * temporary memremap() mapping for the cache flush.
> + */
> +static void sev_clflush_pfn(kvm_pfn_t pfn, size_t size)
> +{
> +	void *va;
> +
> +	if (pfn_valid(pfn)) {
> +		clflush_cache_range(__va(pfn_to_hpa(pfn)), size);
> +		return;
> +	}
> +
> +	va = memremap(pfn_to_hpa(pfn), size, MEMREMAP_WB);
> +	if (WARN_ON_ONCE(!va))

[Severity: High]
Is it safe to use WARN_ON_ONCE(!va) here when memremap() fails?

Since userspace can trigger sev_clflush_pfn() during guest_memfd hole-punching
(via sev_gmem_invalidate()), an unprivileged process might be able to trigger
this warning by inducing memory pressure or vmalloc space exhaustion.

This could result in a host denial of service if panic_on_warn is set.

Should we drop the assertion here to prevent unprivileged userspace from
intentionally triggering host warnings?

> +		return;
> +
> +	clflush_cache_range(va, size);
> +	memunmap(va);
> +}

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/f09647f427565c1789a6979470045aab78661563.1784194290.git.dwmw@amazon.co.uk?part=3

Re: [RFC PATCH 01/11] KVM: selftests: sev_smoke_test: Only run VM types the host offers

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The test silently returns 0 (PASS) if no SEV VM types are offered by KVM, masking the lack of test coverage.
--

--- Patch [1]: [RFC PATCH 01/11] KVM: selftests: sev_smoke_test: Only run VM types the host offers ---
commit c4fa8ea2868d0892d394adecf158fe07893c3980
Author: David Woodhouse <dwmw@amazon.co.uk>

KVM: selftests: sev_smoke_test: Only run VM types the host offers

sev_smoke_test is updated to gate the plain SEV subtest on KVM_CAP_VM_TYPES,
preventing the test from aborting when all SEV ASIDs are assigned to SEV-SNP.
This ensures the test runs only the VM types the host actually offers.

> diff --git a/tools/testing/selftests/kvm/x86/sev_smoke_test.c b/tools/testing/selftests/kvm/x86/sev_smoke_test.c
> index 6b2cbe2a90b7c..bf27b6187afac 100644
> --- a/tools/testing/selftests/kvm/x86/sev_smoke_test.c
> +++ b/tools/testing/selftests/kvm/x86/sev_smoke_test.c
> @@ -247,7 +247,14 @@ int main(int argc, char *argv[])
>  {
>  	TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_SEV));
>  
> -	test_sev_smoke(guest_sev_code, KVM_X86_SEV_VM, 0);
> +	/*
> +	 * Only exercise VM types the host actually offers.  CPUID reporting
> +	 * SEV does not guarantee KVM offers the SEV VM type: when all SEV
> +	 * ASIDs are assigned to SEV-SNP, KVM_X86_SEV_VM is unavailable even
> +	 * though X86_FEATURE_SEV is set.  Gate every type on KVM_CAP_VM_TYPES.
> +	 */
> +	if (kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SEV_VM))
> +		test_sev_smoke(guest_sev_code, KVM_X86_SEV_VM, 0);

[Severity: Medium]
Does this code silently return 0 (PASS) if the host offers no SEV VM types?

If none of the KVM_CAP_VM_TYPES conditions evaluate to true, the program will
fall through to return 0. This masks a lack of test coverage by falsely
indicating a successful test run, rather than properly skipping the test.

>  
>  	if (kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(KVM_X86_SEV_ES_VM))
>  		test_sev_smoke(guest_sev_es_code, KVM_X86_SEV_ES_VM, SEV_POLICY_ES);

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/f09647f427565c1789a6979470045aab78661563.1784194290.git.dwmw@amazon.co.uk?part=1

Re: [RFC PATCH 05/11] iommufd: Look up private-interconnect phys via exporter symbols

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Mapping guest_memfd CPU RAM as uncached MMIO creates memory attribute aliasing, breaking bisection.
--

--- Patch [5]: [RFC PATCH 05/11] iommufd: Look up private-interconnect phys via exporter symbols ---
commit a4bfbccbc028813d0ddd8502c6dc498ce23a16cb
Author: David Woodhouse <dwmw@amazon.co.uk>

iommufd: Look up private-interconnect phys via exporter symbols

This commit widens the dma-buf phys-map dispatch in iommufd to support
additional exporters by looking up a well-known symbol convention. It
initially tries a sample gmem provider before falling back to VFIO PCI.

> diff --git a/drivers/iommu/iommufd/pages.c b/drivers/iommu/iommufd/pages.c
> index 03c8379bbc347..2d4ea41460fdc 100644
> --- a/drivers/iommu/iommufd/pages.c
> +++ b/drivers/iommu/iommufd/pages.c

[ ... ]

> @@ -1470,6 +1470,26 @@ sym_vfio_pci_dma_buf_iommufd_map(struct dma_buf_attachment *attachment,
>  	if (rc != -EOPNOTSUPP)
>  		return rc;
>  
> +	/*
> +	 * Prototype: try the sample gmem provider's dma-buf exporter.  This
> +	 * mirrors the vfio-pci private-interconnect hook, and (like it) is
> +	 * meant to be replaced by a formal negotiated exporter op returning
> +	 * phys for iommufd.
> +	 */
> +	{
> +		extern int gmem_provider_dma_buf_iommufd_map(
> +			struct dma_buf_attachment *, struct phys_vec *);
> +		typeof(&gmem_provider_dma_buf_iommufd_map) gfn;
> +
> +		gfn = symbol_get(gmem_provider_dma_buf_iommufd_map);
> +		if (gfn) {
> +			rc = gfn(attachment, phys);
> +			symbol_put(gmem_provider_dma_buf_iommufd_map);
> +			if (rc != -EOPNOTSUPP)
> +				return rc;

[Severity: High]
Will mapping guest_memfd CPU RAM as uncached MMIO cause memory attribute
aliasing issues here?

When gmem_provider_dma_buf_iommufd_map returns a physical address for RAM,
pfn_reader_fill_dmabuf() will hardcode it as BATCH_MMIO:

drivers/iommu/iommufd/pages.c:pfn_reader_fill_dmabuf() {
    ...
    batch_add_pfn_num(batch, PHYS_PFN(dmabuf->phys.paddr + start),
                      last_index - start_index + 1, BATCH_MMIO);
    ...
}

This causes batch_to_domain() to forcefully strip IOMMU_CACHE and apply
IOMMU_MMIO:

drivers/iommu/iommufd/pages.c:batch_to_domain() {
    ...
    if (batch->kind == BATCH_MMIO) {
        iommu_prot &= ~IOMMU_CACHE;
        iommu_prot |= IOMMU_MMIO;
    }
    ...
}

On architectures like ARM64, this memory attribute aliasing between the
CPU (cacheable) and the IOMMU (uncached) can lead to a loss of coherency,
data corruption, or fatal SErrors.

Since the memory type plumbing is deferred to a later patch, could this
intermediate state introduce a bisection breakage if the sample provider
is utilized?

> +		}
> +	}
> +
>  	if (!IS_ENABLED(CONFIG_VFIO_PCI_DMABUF))
>  		return -EOPNOTSUPP;
>

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/f09647f427565c1789a6979470045aab78661563.1784194290.git.dwmw@amazon.co.uk?part=5

Re: [RFC PATCH 06/11] iommufd: Plumb dma-buf memory-type (RAM vs MMIO) through the phys map

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] Secondary domain attachments map MMIO DMA-bufs with BATCH_CPU_MEMORY (IOMMU_CACHE) instead of BATCH_MMIO (IOMMU_MMIO), breaking DMA routing.
--

--- Patch [6]: [RFC PATCH 06/11] iommufd: Plumb dma-buf memory-type (RAM vs MMIO) through the phys map ---
commit de7dbdc51049f527858115cc709f25b2cee585cb
Author: David Woodhouse <dwmw@amazon.co.uk>

iommufd: Plumb dma-buf memory-type (RAM vs MMIO) through the phys map

This commit plumbs the memory type of a dma-buf (CPU RAM vs MMIO) through
the IOMMU programming path. It adds an is_cpu_ram flag to dmabuf structures
to correctly choose between BATCH_CPU_MEMORY and BATCH_MMIO during mapping.
This prevents silent misrouting of device DMA on AMD-Vi when the physical
memory is RAM instead of a PCI BAR.

> diff --git a/drivers/iommu/iommufd/pages.c b/drivers/iommu/iommufd/pages.c
> index 2d4ea41460fdc..f9b2ae6d7e96c 100644
> --- a/drivers/iommu/iommufd/pages.c
> +++ b/drivers/iommu/iommufd/pages.c

[ ... ]

> @@ -1106,9 +1108,15 @@ static int pfn_reader_fill_dmabuf(struct pfn_reader_dmabuf *dmabuf,
>  	 * always filled using page size aligned PFNs just like the other types.
>  	 * If the dmabuf has been sliced on a sub page offset then the common
>  	 * batch to domain code will adjust it before mapping to the domain.
> +	 *
> +	 * The exporter's memory type (CPU RAM vs MMIO/BAR) selects the batch
> +	 * kind so downstream iommu_map sets IOMMU_CACHE for cache-coherent RAM
> +	 * or IOMMU_MMIO for BAR memory.  The kind was captured at map time by
> +	 * the exporter dispatch.
>  	 */
>  	batch_add_pfn_num(batch, PHYS_PFN(dmabuf->phys.paddr + start),
> -			  last_index - start_index + 1, BATCH_MMIO);
> +			  last_index - start_index + 1,
> +			  dmabuf->is_cpu_ram ? BATCH_CPU_MEMORY : BATCH_MMIO);
>  	return 0;
>  }

[Severity: High]
This is a pre-existing issue, but does attaching a secondary domain map MMIO
DMA-bufs with BATCH_CPU_MEMORY instead of BATCH_MMIO?

When reading PFNs for a secondary domain, pfn_reader_fill_span() routes the
read to batch_from_domain() instead of pfn_reader_fill_dmabuf():

pfn_reader_fill_span() {
	if (span->is_used == 2) {
		/* ... */
		batch_from_domain(
			&pfns->batch, area->storage_domain, area, ...);
		return 0;
	}
}

Then batch_from_domain() extracts PFNs from the already mapped domain and
unconditionally delegates to batch_add_pfn():

batch_from_domain() {
	/* ... */
	phys = iommu_iova_to_phys(domain, iova) - page_offset;
	if (!batch_add_pfn(batch, PHYS_PFN(phys)))
		return;
}

And batch_add_pfn() explicitly hardcodes BATCH_CPU_MEMORY:

static bool batch_add_pfn(struct pfn_batch *batch, unsigned long pfn)
{
	return batch_add_pfn_num(batch, pfn, 1, BATCH_CPU_MEMORY);
}

Could this silently overwrite the original memory type (e.g., BATCH_MMIO)
for secondary domain mappings, causing device DMA to misroute on AMD-Vi
even after this patch?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/f09647f427565c1789a6979470045aab78661563.1784194290.git.dwmw@amazon.co.uk?part=6

Re: [RFC PATCH 07/11] KVM: guest_memfd: Add ops-driven page revocation

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] kvm_gmem_invalidate_range() fails to invalidate GFNs that span multiple memslots, start in an unmapped hole, or exist in non-zero address spaces, leading to partial invalidation and potential Use-After-Free.
--

--- Patch [7]: [RFC PATCH 07/11] KVM: guest_memfd: Add ops-driven page revocation ---
commit 9715bf20df84962c5cc8863e7f4350b3bcb15273
Author: David Woodhouse <dwmw@amazon.co.uk>

KVM: guest_memfd: Add ops-driven page revocation

This commit adds kvm_gmem_invalidate_range() to allow a guest_memfd backing
to zap the guest secondary-MMU mapping for a gfn range. This provides an
implementation-to-KVM path to reclaim pages from a running guest.

> diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
> index 2b277468a12f0..6c13742de0680 100644
> --- a/virt/kvm/guest_memfd.c
> +++ b/virt/kvm/guest_memfd.c

[ ... ]

> @@ -738,6 +738,45 @@ int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args)
>  	return __kvm_gmem_create(kvm, size, flags);
>  }
>  
> +/*
> + * Outbound (implementation -> KVM) revocation: an implementation zaps the
> + * guest's secondary MMU mapping for a gfn range so the next guest access
> + * re-faults through ops->get_pfn().  Mirrors __kvm_gmem_invalidate_start()
> + * for a single bound slot, bracketing the unmap with the mmu_invalidate
> + * window so a racing vCPU fault retries rather than installing a stale
> + * mapping.
> + */
> +void kvm_gmem_invalidate_range(struct kvm *kvm, gfn_t start, gfn_t end)
> +{
> +	struct kvm_memory_slot *slot;
> +	bool flush = false;
> +	int idx;
> +
> +	idx = srcu_read_lock(&kvm->srcu);
> +
> +	slot = gfn_to_memslot(kvm, start);
> +	if (slot) {
> +		struct kvm_gfn_range gfn_range = {
> +			.slot		= slot,
> +			.start		= start,
> +			.end		= min(end, slot->base_gfn + slot->npages),
> +			.may_block	= true,
> +			.attr_filter	= KVM_FILTER_SHARED | KVM_FILTER_PRIVATE,
> +		};

[Severity: High]
Does this code fully invalidate the requested range if it spans multiple
memslots or starts in an unmapped hole?

By looking up only the memslot for the start GFN and clamping the end to
slot->base_gfn + slot->npages, it seems any remaining GFNs in the requested
span will be skipped if the range crosses a memslot boundary.

Additionally, if the start GFN happens to fall in an unmapped hole,
gfn_to_memslot() returns NULL and the entire invalidation is bypassed, even
if there are valid mapped memslots further along in the range.

Finally, gfn_to_memslot() only checks Address Space 0. Could this leave
stale SPTEs in alternate address spaces (like SMM), allowing the guest to
access reclaimed host memory?

> +
> +		KVM_MMU_LOCK(kvm);
> +		kvm_mmu_invalidate_start(kvm);
> +		flush = kvm_mmu_unmap_gfn_range(kvm, &gfn_range);
> +		if (flush)
> +			kvm_flush_remote_tlbs(kvm);
> +		kvm_mmu_invalidate_end(kvm);
> +		KVM_MMU_UNLOCK(kvm);
> +	}
> +
> +	srcu_read_unlock(&kvm->srcu, idx);
> +}

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/f09647f427565c1789a6979470045aab78661563.1784194290.git.dwmw@amazon.co.uk?part=7

Re: [RFC PATCH 09/11] selftests/kvm: gmem_provider KVM-only tests

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] The test does not check for partial completion of KVM_SEV_SNP_LAUNCH_UPDATE, leading to potentially silent incomplete memory initialization.
- [Medium] The test fails to assert KVM_EXIT_MEMORY_FAULT on the revoke path because the validation is placed in an unreachable 'else' branch.
--

--- Patch [9]: [RFC PATCH 09/11] selftests/kvm: gmem_provider KVM-only tests ---
commit 93ae3986a0340a4d14084b20d7839ed583b0c01b
Author: David Woodhouse <dwmw@amazon.co.uk>

selftests/kvm: gmem_provider KVM-only tests

Three selftests that exercise the guest_memfd provider ABI purely from
the KVM side (no iommufd, no assigned devices). These tests verify the
live-update survivability, hugepage mappings, and memory revocation.

> diff --git a/tools/testing/selftests/kvm/x86/gmem_provider_revoke_test.c b/tools/testing/selftests/kvm/x86/gmem_provider_revoke_test.c
> index 0000000000000..7e711ea526748 100644
> --- /dev/null
> +++ b/tools/testing/selftests/kvm/x86/gmem_provider_revoke_test.c
[ ... ]
> +	/* 2) Revoke: mark absent and zap the guest NPT (provider->KVM). */
> +	req = (struct gmem_provider_present){ .offset = 0, .len = 4096, .present = 0 };
> +	r = ioctl(gmem_fd, GMEM_PROVIDER_SET_PRESENT, &req);
> +	TEST_ASSERT(!r, "revoke ioctl failed, errno %d", errno);
> +
> +	/* 3) Guest re-reads -> re-fault into absent get_pfn -> must NOT see MAGIC. */
> +	r = _vcpu_run(vcpu);
> +	if (r) {
> +		pr_info("revoked: vcpu_run failed (errno %d) -- guest faulted, as expected\n",
> +			errno);
> +	} else {
> +		TEST_ASSERT(vcpu->run->exit_reason == KVM_EXIT_MEMORY_FAULT,
> +			    "revoked: expected fault or KVM_EXIT_MEMORY_FAULT, got %u (%s)",
> +			    vcpu->run->exit_reason,
> +			    exit_reason_str(vcpu->run->exit_reason));
> +		pr_info("revoked: KVM_EXIT_MEMORY_FAULT as expected\n");
> +	}

[Severity: Medium]
Since KVM_EXIT_MEMORY_FAULT always causes the KVM_RUN ioctl to return -1
with errno EFAULT, won't _vcpu_run(vcpu) return a non-zero value here?

If so, this execution path would take the if (r) branch, completely bypassing
the exit reason validation in the else block. Should this check be moved or
adjusted to properly assert that the exit reason is indeed
KVM_EXIT_MEMORY_FAULT?

> +
> +	/* 4) Restore: mark present again. */
> +	req.present = 1;
> +	r = ioctl(gmem_fd, GMEM_PROVIDER_SET_PRESENT, &req);
> +	TEST_ASSERT(!r, "restore ioctl failed, errno %d", errno);
> +

[ ... ]
> diff --git a/tools/testing/selftests/kvm/x86/gmem_provider_test.c b/tools/testing/selftests/kvm/x86/gmem_provider_test.c
> index 0000000000000..d7caa11616dfe 100644
> --- /dev/null
> +++ b/tools/testing/selftests/kvm/x86/gmem_provider_test.c
[ ... ]
> +	struct kvm_sev_snp_launch_start start = { .policy = 0x30000 };
> +	ret = sev_ioctl(vm_fd, sev_fd, KVM_SEV_SNP_LAUNCH_START, &start);
> +	if (ret) { perror("SNP_LAUNCH_START"); goto out; }
> +
> +	printf("[VM%d] SNP_LAUNCH_UPDATE (code + zero, %luMB)\n",
> +	       vm_num, GUEST_MEM_SIZE >> 20);
> +	struct kvm_sev_snp_launch_update update = {
> +		.gfn_start = 0,
> +		.uaddr = (uint64_t)(unsigned long)src,
> +		.len = PAGE_SIZE_4K,
> +		.type = KVM_SEV_SNP_PAGE_TYPE_NORMAL,
> +	};
> +	ret = sev_ioctl(vm_fd, sev_fd, KVM_SEV_SNP_LAUNCH_UPDATE, &update);
> +	if (ret) { perror("SNP_LAUNCH_UPDATE code"); goto out; }

[Severity: Medium]
If a signal interrupts the KVM_SEV_SNP_LAUNCH_UPDATE ioctl, KVM might
return 0 but leave update.len > 0 indicating a partial completion.

Does this code need a loop to check update.len and retry the remaining
length to prevent silent partial initialization?

> +
> +	struct kvm_sev_snp_launch_update update_zero = {
> +		.gfn_start = 1,
> +		.uaddr = (uint64_t)(unsigned long)(src + PAGE_SIZE_4K),
> +		.len = GUEST_MEM_SIZE - PAGE_SIZE_4K,
> +		.type = KVM_SEV_SNP_PAGE_TYPE_ZERO,
> +	};
> +	ret = sev_ioctl(vm_fd, sev_fd, KVM_SEV_SNP_LAUNCH_UPDATE, &update_zero);
> +	if (ret) { perror("SNP_LAUNCH_UPDATE zero"); goto out; }

[Severity: Medium]
Similarly, could a signal interrupting this update_zero ioctl cause a partial
completion here as well? If the return value is 0 but update_zero.len is
still greater than 0, the rest of the 16MB guest memory would be left
uninitialized, potentially causing flaky test failures later.

> +
> +	struct kvm_sev_snp_launch_finish finish = { 0 };
> +	ret = sev_ioctl(vm_fd, sev_fd, KVM_SEV_SNP_LAUNCH_FINISH, &finish);

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/f09647f427565c1789a6979470045aab78661563.1784194290.git.dwmw@amazon.co.uk?part=9

Re: [RFC PATCH 10/11] selftests/kvm: gmem_provider iommufd tests

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 5 potential issue(s) to consider:
- [Medium] The NVMe DMA test hardcodes the Amazon PCI Vendor ID (0x1d0f), causing false failures on generic hardware.
- [Medium] Missing compiler memory barrier before ringing the NVMe Submission Queue doorbell allows dangerous compiler reordering.
- [Medium] gmem_provider_iommufd_test fails to exercise the KVM guest_memfd private memory path because it does not set the KVM_MEMORY_ATTRIBUTE_PRIVATE attribute.
- [Medium] The test claims to confirm a fresh IOMMU_IOAS_MAP_FILE succeeds after restoring the page, but does not actually issue the ioctl.
- [Low] Comment incorrectly claims the NVMe completion phase bit is inverted, contradicting the code and the NVMe specification.
--

--- Patch [10]: [RFC PATCH 10/11] selftests/kvm: gmem_provider iommufd tests ---
commit 104877080caa9ea2c6214f716d67d2b54448bb37
Author: David Woodhouse <dwmw@amazon.co.uk>
Date:   Thu Jul 16 09:31:30 2026 +0000

    selftests/kvm: gmem_provider iommufd tests
    
    Three selftests that exercise the guest_memfd provider ABI from the
    iommufd side, going progressively closer to a real assigned-device
    setup.

> diff --git a/tools/testing/selftests/kvm/x86/gmem_provider_iommufd_test.c b/tools/testing/selftests/kvm/x86/gmem_provider_iommufd_test.c
> new file mode 100644
> index 0000000000000..0b2fc097b5a91
> --- /dev/null
> +++ b/tools/testing/selftests/kvm/x86/gmem_provider_iommufd_test.c

[ ... ]

> +	hva = mmap(NULL, DATA_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, gmem_fd, 0);
> +	TEST_ASSERT(hva != MAP_FAILED,
> +		    "provider mmap failed errno=%d", errno);
> +	r = __vm_set_user_memory_region2(vm, DATA_SLOT, KVM_MEM_GUEST_MEMFD,
> +					 DATA_GPA, DATA_SIZE, hva, gmem_fd, 0);
> +	TEST_ASSERT(!r, "KVM_SET_USER_MEMORY_REGION2 failed r=%d errno=%d", r, errno);
> +	virt_map(vm, DATA_GPA, DATA_GPA, 1);
> +
> +	/*
> +	 * 3) iommufd side: allocate IOAS

[Severity: Medium]
Does main() actually exercise the KVM guest_memfd private memory path?

Because the KVM_MEMORY_ATTRIBUTE_PRIVATE attribute is never set on this
GPA via vm_mem_set_private(), KVM will fault on the shared memory HVA.

Does this completely bypass the KVM guest_memfd logic, with the guest write
reaching the provider fd via the HVA mmap path instead?

[ ... ]

> diff --git a/tools/testing/selftests/kvm/x86/gmem_provider_nvme_dma_test.c b/tools/testing/selftests/kvm/x86/gmem_provider_nvme_dma_test.c
> new file mode 100644
> index 0000000000000..8c8a4b2e8ac96
> --- /dev/null
> +++ b/tools/testing/selftests/kvm/x86/gmem_provider_nvme_dma_test.c

[ ... ]

> +	/* Identify Controller SQE at slot 0. */
> +	{
> +		uint32_t *s = (uint32_t *)sq_buf;
> +		memset(s, 0, 64);
> +		s[0]  = 0x00010006;	/* CID=1 << 16 | OPC=Identify */
> +		s[1]  = 0;		/* NSID */
> +		/* PRP1 (offset 24 in SQE = dword 6) */
> +		s[6]  = (uint32_t)IOVA_DST;
> +		s[7]  = (uint32_t)(IOVA_DST >> 32);
> +		s[8]  = 0;
> +		s[9]  = 0;		/* PRP2 */
> +		s[10] = 0x01;		/* CDW10: CNS = Identify Controller */
> +	}
> +
> +	/* Ring SQ0 tail doorbell. */
> +	wr32(bar, NVME_SQ0TDBL, 1);

[Severity: Medium]
Is there a risk of the compiler reordering the SQE initialization after
the doorbell ring in main()?

Since the SQE is prepared via non-volatile stores and the doorbell is
written via a volatile cast, the C standard permits reordering
non-volatile stores after volatile stores. Could this result in the NVMe
controller processing an incomplete or zeroed command?

Would it be safer to add a compiler barrier before writing the doorbell?

[ ... ]

> +	/* Provider region now holds Identify Controller data.  VID/SSVID at bytes 0..3
> +	 * must equal the device's PCI vendor + subsystem vendor (both are Amazon here). */
> +	{
> +		uint16_t vid  = ((uint16_t *)provider_hva)[0];
> +		uint16_t ssvid = ((uint16_t *)provider_hva)[1];
> +		uint8_t first = ((uint8_t *)provider_hva)[0];
> +		uint8_t poison_still = ((uint8_t *)provider_hva)[8];
> +		char mn[41] = {};
> +		memcpy(mn, (char *)provider_hva + 24, 40);
> +		fprintf(stderr, "MARKER: read done; byte0=0x%02x byte8=0x%02x\n",
> +			first, poison_still);
> +		fflush(stderr);
> +		pr_info("Identify Controller in provider: VID=0x%04x SSVID=0x%04x MN='%s'\n",
> +			vid, ssvid, mn);
> +		fflush(stdout);
> +		TEST_ASSERT(vid == 0x1d0f,
> +			    "unexpected VID 0x%04x in provider region -- Identify DMA landed elsewhere?\n"
> +			    "  poison=0x%02x byte0=0x%02x (poison intact=DMA didn't land here)",
> +			    vid, poison_still, first);

[Severity: Medium]
Does main() hardcode the Amazon PCI Vendor ID here?

If this test is run on a non-Amazon NVMe drive, won't this assertion
cause it to fail on generic hardware?

Should we fetch the PCI Vendor ID from the config space instead of
hardcoding it to 0x1d0f?

[ ... ]

> +		/* Issue a second Identify Controller aimed at the revoked IOVA.
> +		 * Use a new CID and place the SQE in slot 1; the SQ ring will
> +		 * wrap for larger runs but we only ever use slots 0 and 1.
> +		 */
> +		memset(&s[16], 0, 64);
> +		s[16] = 0x00020006;			/* CID=2 << 16 | OPC=Identify */
> +		s[16 + 6] = (uint32_t)IOVA_DST;
> +		s[16 + 7] = (uint32_t)(IOVA_DST >> 32);
> +		s[16 + 10] = 0x01;			/* CNS = Identify Controller */
> +
> +		wr32(bar, NVME_SQ0TDBL, 2);

[Severity: Medium]
Similar to the first command submission in main(), does this need a
compiler barrier before the doorbell ring to prevent the compiler from
sinking the SQE initialization?

> +		pr_info("Identify Controller (revoked target) submitted (SQ0 tail=2)\n");
> +
> +		/* Poll for CQE at slot 1 with phase bit inverted from first pass. */
> +		for (i = 0; i < 5000; i++) {
> +			if ((c[4 + 3] & 0x10000))	/* CQE 1 P bit */

[Severity: Low]
Is this comment accurate?

Since this is only the second command in a 64-entry queue, the controller
is still on its first pass. The phase bit remains 1 and does not invert,
which matches the code's correct check for a 1 bit.

[ ... ]

> +		/* Restore and confirm a fresh IOMMU_IOAS_MAP_FILE succeeds. */
> +		rev.present = 1;
> +		ret = ioctl(gmem_fd, GMEM_PROVIDER_SET_PRESENT, &rev);
> +		TEST_ASSERT(ret == 0, "SET_PRESENT(restore) errno=%d", errno);
> +		pr_info("Restore + revoke round trip complete\n");
> +	}

[Severity: Medium]
The comment in main() mentions confirming a fresh IOMMU_IOAS_MAP_FILE
succeeds, but does the test actually attempt to issue this ioctl after
restoring the page?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/f09647f427565c1789a6979470045aab78661563.1784194290.git.dwmw@amazon.co.uk?part=10

Re: [RFC PATCH 08/11] samples/kvm: Add guest_memfd backing sample

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 14 potential issue(s) to consider:
- [Critical] Missing file type validation for `setup.kvm_fd` allows arbitrary kernel memory corruption.
- [Critical] `gmem_release()` frees memory while it is still mapped into the guest, causing host physical memory corruption.
- [High] Unprotected access to `info->kvm` leads to Use-After-Free and data races.
- [High] Non-atomic bitmap operations in `GMEM_PROVIDER_SET_PRESENT` corrupt memory revocation state.
- [High] `gmem_bind` tracks only a single memslot binding, preventing memory revocation for previously bound slots.
- [High] Integer underflow in revocation range causes out-of-bounds KVM invalidation.
- [High] `gmem_max_order()` ignores the absent bitmap, allowing KVM to install hugepage mappings over revoked pages.
- [High] `gmem_mmap()` does not check `info->mmap_capable`, allowing host access to hardware-encrypted VM memory.
- [High] `gmem_unbind()` fails to call `psmash()`, leaving 2M pages permanently stuck in the private state on SEV-SNP.
- [High] The `psmash` operation is completely skipped in `gmem_bind` if the starting physical address is not 2M aligned.
- [High] The provider does not pin its module, allowing it to be unloaded while guest_memfd instances remain active.
- [High] Integer truncation in `bitmap_zalloc()` leads to an unprivileged kernel heap buffer overflow.
- [Medium] Leaked provider file reference in `gmem_provider_get_dmabuf` error path.
- [Medium] DMAbuf exporter leaks its initial kref and fails to wait for mappings to be torn down.
--

--- Patch [8]: [RFC PATCH 08/11] samples/kvm: Add guest_memfd backing sample ---
commit 7bfeb9f519ccce1291a261c36607819a68c027ce
Author: David Woodhouse <dwmw@amazon.co.uk>
Subject: samples/kvm: Add guest_memfd backing sample

This commit adds a sample guest_memfd provider module for KVM. It
demonstrates page-less backing from a fixed physical range or a
CMA-allocated contiguous region. It also includes page revocation
via a bitmap and a dma-buf exporter.

> diff --git a/samples/kvm/gmem_provider.c b/samples/kvm/gmem_provider.c
> new file mode 100644
> index 0000000000000..e30580fecd166
> --- /dev/null
> +++ b/samples/kvm/gmem_provider.c
> @@ -0,0 +1,651 @@
[ ... ]
> +static int gmem_max_order(struct gmem_info *info, gfn_t gfn, unsigned long index)
> +{
> +	unsigned long pfn = info->base_pfn + index;
> +	unsigned long remaining = info->npages - index;
> +	unsigned int pud_order = PUD_SHIFT - PAGE_SHIFT;
> +	unsigned int pmd_order = PMD_SHIFT - PAGE_SHIFT;
> +
> +	if (IS_ALIGNED(pfn, 1UL << pud_order) &&
> +	    IS_ALIGNED(gfn, 1UL << pud_order) &&
> +	    remaining >= (1UL << pud_order))
> +		return pud_order;

[Severity: High]
Does gmem_max_order() account for pages marked absent?

When KVM queries the maximum mapping order for a present faulting page,
gmem_max_order() checks alignment and remaining length, but does not check
if any of the subsequent pages within the hugepage range are marked absent
in info->absent.

If a subset of pages in a 2M or 1G range is revoked but the faulting page
is present, could KVM install a hugepage mapping that grants the guest
unauthorized access to the revoked pages within that block?

[ ... ]
> +static int gmem_bind(struct file *file, struct kvm *kvm,
> +		     struct kvm_memory_slot *slot, loff_t offset)
> +{
[ ... ]
> +	/* Record the binding so the revoke ioctl can translate offset -> gfn. */
> +	info->base_gfn = slot->base_gfn;
> +	info->pgoff = start;

[Severity: High]
Does gmem_bind() only track a single memslot binding?

KVM allows a single provider fd to back multiple memslots. However,
gmem_bind() overwrites info->base_gfn and info->pgoff with the new
slot's coordinates.

During GMEM_PROVIDER_SET_PRESENT, KVM is only asked to invalidate the single,
most recently bound GFN range. Older slots would be ignored and their guest
NPT/EPT mappings never zapped. Can this result in a complete bypass of the
page revocation mechanism for previously bound slots?

> +
> +#if IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT)
> +	/*
> +	 * Reset the RMP for the range to 4K shared so a (new) SEV-SNP VM can
> +	 * transition it to private and re-encrypt it.  PSMASH any 2M entries
> +	 * first.  Harmless on non-SNP hosts, where these return -ENODEV.
> +	 */
> +	{
> +		unsigned long i;
> +
> +		for (i = 0; i < slot->npages; i += PTRS_PER_PMD) {
> +			unsigned long pfn = info->base_pfn + start + i;
> +
> +			if (IS_ALIGNED(pfn, PTRS_PER_PMD))
> +				psmash(pfn);
> +		}

[Severity: High]
Will the psmash() operation be skipped if the start address is unaligned?

The loop advances i by PTRS_PER_PMD (which is 512) and checks if the combined
pfn is perfectly aligned to a 2M boundary.

If info->base_pfn + start is not 2M-aligned to begin with, adding 512
guarantees that every subsequent PFN checked will also be unaligned.
psmash() would never be called for any 2M boundaries inside the range.
Could this cause subsequent rmp_make_shared() calls to fail?

> +		for (i = 0; i < slot->npages; i++) {
> +			unsigned long pfn = info->base_pfn + start + i;
> +			int ret = rmp_make_shared(pfn, PG_LEVEL_4K);
> +
> +			if (ret && ret != -ENODEV)
> +				pr_info_once("gmem_provider: rmp_make_shared(0x%lx) = %d\n",
> +					     pfn, ret);
> +		}
> +	}
> +#endif
> +
> +	/* Claim (or, on re-bind, transfer) VM ownership; pins the VM. */
> +	if (info->kvm != kvm) {
> +		kvm_get_kvm(kvm);
> +		if (info->kvm)
> +			kvm_put_kvm(info->kvm);
> +		info->kvm = kvm;
> +	}

[Severity: High]
Does this code handle concurrent bindings safely?

In gmem_bind(), info->kvm is read and modified without any locking. If a
single provider file is concurrently bound to multiple VMs (or the same VM
multiple times), multiple threads could race, reading the same old info->kvm
pointer and both calling kvm_put_kvm().

This could lead to a double-put and Use-After-Free of the KVM instances.
Additionally, gmem_fd_ioctl() accesses info->kvm without locks while it
might be concurrently updated or freed. Could we add synchronization to
protect info->kvm?

[ ... ]
> +static void gmem_unbind(struct file *file, struct kvm *kvm,
> +			struct kvm_memory_slot *slot)
> +{
> +#if IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT)
> +	struct gmem_info *info = to_gmem_info(file);
> +	unsigned long start = slot->gmem.pgoff;
> +	unsigned long i;
> +
> +	for (i = 0; i < slot->npages; i++) {
> +		unsigned long pfn = info->base_pfn + start + i;
> +
> +		rmp_make_shared(pfn, PG_LEVEL_4K);
> +	}
> +#endif
> +}

[Severity: High]
Is a call to psmash() required before rmp_make_shared() here?

On SEV-SNP hosts, converting a 2M private page back to the shared state
using rmp_make_shared() with PG_LEVEL_4K fails unless the 2M RMP entry is
first shattered using psmash().

gmem_unbind() omits the psmash() step and ignores the return value of
rmp_make_shared(). This could leave 2M pages permanently marked as private.
When these CMA pages are freed and reused by the host allocator, wouldn't
accessing them crash the host?

[ ... ]
> +static void gmem_release(struct file *file)
> +{
> +	struct gmem_info *info = to_gmem_info(file);
> +
> +	if (info->kvm)
> +		kvm_put_kvm(info->kvm);
> +	if (info->cma_pages)
> +		free_contig_range(info->base_pfn, info->npages);
> +	bitmap_free(info->absent);
> +	kfree(info);
> +}

[Severity: Critical]
Does gmem_release() free memory while it is still mapped into the guest?

When the provider fd is closed, gmem_release() frees info->cma_pages and
the info struct. However, it does not appear to zap the guest's NPT/EPT
page tables or clear the KVM slot tracking pointer.

This leaves the guest with full read/write access to the freed CMA pages
after they have been reallocated to other kernel subsystems. When KVM later
destroys the memslot, it could also dereference the dangling slot->gmem.file
pointer. Should the mappings be explicitly torn down before freeing?

[ ... ]
> +static int gmem_mmap(struct file *file, struct vm_area_struct *vma)
> +{
> +	struct gmem_info *info = to_gmem_info(file);
> +	unsigned long npages = vma_pages(vma);
> +
> +	if (vma->vm_pgoff + npages > info->npages)
> +		return -EINVAL;
> +
> +	/* Page-less backing: map raw PFNs, not folios. */
> +	vm_flags_set(vma, VM_PFNMAP | VM_IO | VM_DONTEXPAND | VM_DONTDUMP);
> +	return remap_pfn_range(vma, vma->vm_start, info->base_pfn + vma->vm_pgoff,
> +			       npages << PAGE_SHIFT, vma->vm_page_prot);
> +}

[Severity: High]
Should gmem_mmap() validate the mmap_capable policy?

gmem_bind() refuses to bind to encrypted VMs if info->mmap_capable is set,
in order to protect the host. However, gmem_mmap() maps the memory to
userspace without verifying info->mmap_capable.

Userspace could clear GMEM_PROVIDER_FLAG_MMAP_CAPABLE during setup,
successfully bind to a SEV-SNP/TDX VM, and then call mmap() on the fd
anyway. Could this allow host access to hardware-encrypted memory and
cause a machine check exception or RMP violation?

[ ... ]
> +/* Called with info->dmabufs_lock held on the revoke path. */
> +static void gmem_dma_buf_revoke_all(struct gmem_info *info)
> +{
> +	struct gmem_dmabuf *priv;
> +
> +	list_for_each_entry(priv, &info->dmabufs, list) {
> +		dma_resv_lock(priv->dmabuf->resv, NULL);
> +		if (!priv->revoked) {
> +			priv->revoked = true;
> +			dma_buf_invalidate_mappings(priv->dmabuf);
> +		}
> +		dma_resv_unlock(priv->dmabuf->resv);
> +	}
> +}

[Severity: Medium]
Should the revocation wait for the completion of mapping teardowns?

In gmem_dma_buf_revoke_all(), dma_buf_invalidate_mappings() is called, but
it doesn't wait on priv->comp. This leaves the completion machinery unused
and leaks the kref initialized in gmem_provider_get_dmabuf().

Should we properly wait for the mappings to be torn down and drop the
reference?

[ ... ]
> +static int gmem_provider_get_dmabuf(struct file *file)
> +{
> +	struct gmem_info *info = to_gmem_info(file);
> +	DEFINE_DMA_BUF_EXPORT_INFO(exp_info);
> +	struct gmem_dmabuf *priv;
> +	int fd;
> +
> +	priv = kzalloc(sizeof(*priv), GFP_KERNEL);
> +	if (!priv)
> +		return -ENOMEM;
> +
> +	priv->info = info;
> +	priv->provider_file = get_file(file);
> +	priv->phys.paddr = (u64)info->base_pfn << PAGE_SHIFT;
> +	priv->phys.len = (u64)info->npages << PAGE_SHIFT;
> +	kref_init(&priv->kref);
> +	init_completion(&priv->comp);
> +	INIT_LIST_HEAD(&priv->list);
> +
> +	exp_info.ops = &gmem_dma_buf_ops;
> +	exp_info.size = priv->phys.len;
> +	exp_info.flags = O_RDWR;
> +	exp_info.priv = priv;
> +
> +	priv->dmabuf = dma_buf_export(&exp_info);
> +	if (IS_ERR(priv->dmabuf)) {
> +		fd = PTR_ERR(priv->dmabuf);
> +		kfree(priv);
> +		return fd;
> +	}

[Severity: Medium]
Will this error path leak the provider file reference?

Earlier in gmem_provider_get_dmabuf(), get_file(file) is called to increment
the reference count. If dma_buf_export() fails, the function frees priv and
returns an error, but it doesn't appear to call fput(priv->provider_file).

Could userspace trigger this intentionally (e.g., by hitting file descriptor
limits) to permanently leak the file and its associated CMA memory?

[ ... ]
> +static long gmem_fd_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
> +{
> +	struct gmem_info *info = to_gmem_info(file);
> +	struct gmem_provider_present p;
> +	unsigned long start_index, end_index;
[ ... ]
> +	if (p.present) {
> +		/* Restore: next guest fault calls get_pfn() and re-maps. */
> +		bitmap_clear(info->absent, start_index, end_index - start_index);
> +	} else {
> +		/* Revoke: mark absent, then zap the guest NPT/EPT for the range. */
> +		bitmap_set(info->absent, start_index, end_index - start_index);

[Severity: High]
Are these bitmap operations safe against concurrent updates?

In gmem_fd_ioctl(), GMEM_PROVIDER_SET_PRESENT uses bitmap_set() and
bitmap_clear() without synchronization. These functions perform non-atomic
read-modify-write operations on the underlying unsigned long elements.

If userspace concurrently revokes or restores different page ranges that fall
within the same bitmap word, won't the updates race and clobber each other,
potentially allowing the guest unauthorized access?

> +		if (info->kvm)
> +			kvm_gmem_invalidate_range(info->kvm,
> +				info->base_gfn + start_index - info->pgoff,
> +				info->base_gfn + end_index - info->pgoff);

[Severity: High]
Can this arithmetic underflow if start_index < info->pgoff?

If userspace specifies a revocation range that starts before the offset where
the slot was bound (start_index < info->pgoff), the subtraction here will
underflow to a massive unsigned value.

kvm_gmem_invalidate_range() would then use this bogus GFN range, fail to
find the correct slot, and skip the invalidation entirely, leaving revoked
pages mapped in the guest. Should we bounds-check start_index against
info->pgoff?

[ ... ]
> +static long gmem_ctl_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
> +{
> +	struct gmem_provider_setup setup;
> +	struct gmem_info *info;
> +	struct file *kvm_file;
> +	struct page *pages = NULL;
> +	struct kvm *kvm;
> +	unsigned long npages;
> +	int fd, ret;
> +
> +	if (cmd != GMEM_PROVIDER_SETUP)
> +		return -ENOTTY;
> +
> +	if (copy_from_user(&setup, (void __user *)arg, sizeof(setup)))
> +		return -EFAULT;
> +	if (setup.flags & ~GMEM_PROVIDER_FLAG_MMAP_CAPABLE)
> +		return -EINVAL;
> +
> +	kvm_file = fget(setup.kvm_fd);
> +	if (!kvm_file)
> +		return -EBADF;
> +	kvm = kvm_file->private_data;

[Severity: Critical]
Is it possible for userspace to provide a file descriptor that is not a KVM
VM file?

Looking at gmem_ctl_ioctl(), the code calls fget() on setup.kvm_fd and blindly
dereferences kvm_file->private_data. If an unprivileged user passes an
arbitrary file descriptor (like an eventfd or socket), this would cast
its private_data to a struct kvm *.

This could lead to arbitrary kernel memory corruption when kvm_get_kvm()
subsequently performs an atomic increment at the offset of users_count on
that arbitrary data structure. Should we verify the file's operations
(e.g., checking kvm_file->f_op) before casting?

[ ... ]
> +	info->absent = bitmap_zalloc(info->npages, GFP_KERNEL);

[Severity: High]
Can info->npages be truncated during bitmap allocation?

info->npages is a 64-bit unsigned long. If the module is loaded with a large
external addr=/len= range (e.g., 16TB), info->npages exceeds 32 bits.
bitmap_zalloc() takes an unsigned int for nbits, which would silently
truncate this value, allocating a smaller bitmap.

An unprivileged user could then call GMEM_PROVIDER_SET_PRESENT with a large
offset that passes the 64-bit bounds check, causing bitmap_set() to write
far beyond the truncated allocation. Does this allow a kernel heap buffer
overflow?

[ ... ]
> +	fd = anon_inode_getfd("[gmem-provider]", &kvm_gmem_fops,
> +			      &info->backing, O_RDWR | O_CLOEXEC);

[Severity: High]
Is the provider module pinned when the file descriptor is created?

anon_inode_getfd() creates the fd using &kvm_gmem_fops, which belongs to
the KVM module. The provider module does not take a reference to itself.

If userspace closes the control fd (/dev/gmem_provider) but keeps the
provider fd open, could the gmem_provider module be unloaded via rmmod?
Subsequent VM page faults or ioctls would then call into the unloaded
module's gmem_ops, potentially causing a kernel panic.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/f09647f427565c1789a6979470045aab78661563.1784194290.git.dwmw@amazon.co.uk?part=8

Re: [RFC PATCH 11/11] samples/kvm, selftests/kvm: Allow the gmem_provider NVMe DMA test on arm64

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 6 potential issue(s) to consider:
- [Critical] The gmem_provider sample module blindly trusts the `setup.kvm_fd` file descriptor, leading to type confusion and local privilege escalation.
- [High] A data race on the `info->kvm` pointer between `gmem_bind` and `gmem_fd_ioctl` allows a concurrent binding to drop the VM reference while it is actively being used, leading to a Use-After-Free.
- [Medium] The NVMe DMA selftest hardcodes the Amazon PCI Vendor ID (0x1d0f), causing it to fail on non-Amazon NVMe devices.
- [Medium] The selftest implicitly assumes a 4KB system page size, causing IOMMU mapping failures on architectures with larger page sizes.
- [Medium] The selftest bypasses architecture-specific I/O accessors and omits mandatory memory barriers, causing data races on weakly-ordered architectures.
- [Medium] The selftest performs raw, native-endian memory accesses on strictly little-endian PCI configuration space and NVMe registers, causing corruption on big-endian architectures.
--

--- Patch [11]: [RFC PATCH 11/11] samples/kvm, selftests/kvm: Allow the gmem_provider NVMe DMA test on arm64 ---
commit fc0ebbb277714921db32af720eb2c5783d061ea0
Author: David Woodhouse <dwmw@amazon.co.uk>

samples/kvm, selftests/kvm: Allow the gmem_provider NVMe DMA test on arm64

The sample provider and the NVMe DMA selftest have no x86 specifics.
Drop the X86_64 gate on the sample's Kconfig (arm64 also has
KVM_GUEST_MEMFD; the SEV RMP reset is already guarded by
CONFIG_AMD_MEM_ENCRYPT) and move the NVMe DMA selftest from
TEST_GEN_PROGS_x86 to TEST_GEN_PROGS_COMMON so it also builds on
architectures with an IOMMU (arm64 SMMU-v3, ...).

> diff --git a/samples/Kconfig b/samples/Kconfig
> index 3482659638b5b..d26a03dea0728 100644
> --- a/samples/Kconfig
> +++ b/samples/Kconfig
> @@ -328,7 +328,7 @@ source "samples/damon/Kconfig"
>  
>  config SAMPLE_KVM_GMEM_PROVIDER
>  	tristate "Build sample guest_memfd provider -- loadable module only"
> -	depends on KVM_GUEST_MEMFD && X86_64 && m
> +	depends on KVM_GUEST_MEMFD && (X86_64 || ARM64) && m

[Severity: Critical]
Is it safe to blindly trust the kvm_fd here?

In gmem_ctl_ioctl(), the user-provided setup.kvm_fd is resolved and used
like this:

samples/kvm/gmem_provider.c:gmem_ctl_ioctl() {
    ...
	kvm_file = fget(setup.kvm_fd);
	if (!kvm_file)
		return -EBADF;
	kvm = kvm_file->private_data;
	if (!kvm) {
		fput(kvm_file);
		return -EINVAL;
	}
	kvm_get_kvm(kvm);
    ...
}

If a user passes an arbitrary file descriptor, it could lead to type confusion
since we don't verify the file's operations before extracting private_data and
performing an out-of-bounds refcount_inc(&kvm->users_count).

[Severity: High]
Can gmem_fd_ioctl() safely use info->kvm without synchronization?

In gmem_fd_ioctl(), info->kvm is used without holding a lock:

samples/kvm/gmem_provider.c:gmem_fd_ioctl() {
    ...
		if (info->kvm)
			kvm_gmem_invalidate_range(info->kvm,
				info->base_gfn + start_index - info->pgoff,
				info->base_gfn + end_index - info->pgoff);
    ...
}

Concurrently, gmem_bind() can swap info->kvm and drop the underlying
reference:

samples/kvm/gmem_provider.c:gmem_bind() {
    ...
	if (info->kvm != kvm) {
		kvm_get_kvm(kvm);
		if (info->kvm)
			kvm_put_kvm(info->kvm);
		info->kvm = kvm;
	}
    ...
}

Could this concurrent binding drop the VM reference to zero while it is being
accessed by the ioctl, leading to a use-after-free?

[ ... ]

> diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm
> index 7f46843e7428f..12004a487c326 100644
> --- a/tools/testing/selftests/kvm/Makefile.kvm
> +++ b/tools/testing/selftests/kvm/Makefile.kvm
> @@ -58,6 +58,7 @@ TEST_PROGS_x86 += x86/nx_huge_pages_test.sh
>  # Compiled test targets valid on all architectures with libkvm support
>  TEST_GEN_PROGS_COMMON = demand_paging_test
>  TEST_GEN_PROGS_COMMON += dirty_log_test
> +TEST_GEN_PROGS_COMMON += gmem_provider_nvme_dma_test

[Severity: Medium]
Will this regression cause the test to fail on host hardware that doesn't use
an Amazon NVMe device?

The test hardcodes the Amazon PCI Vendor ID:

tools/testing/selftests/kvm/gmem_provider_nvme_dma_test.c:main() {
    ...
		TEST_ASSERT(vid == 0x1d0f,
			    "unexpected VID 0x%04x in provider region -- Identify DMA landed elsewhere?\n"
    ...
}

Should the expected VID be read dynamically from offset 0x00 instead?

[Severity: Medium]
Does the implicit 4KB system page size assumption introduce a regression on
architectures that commonly use larger page sizes, such as arm64?

The test uses hardcoded 4KB offsets and lengths for IOMMU mapping:

tools/testing/selftests/kvm/gmem_provider_nvme_dma_test.c:main() {
    ...
	TEST_ASSERT(!ioas_map_user(iommufd_fd, alloc.out_ioas_id, sq_buf, IOVA_SQ, 0x1000),
		    "IOAS_MAP(sq) errno=%d", errno);
	TEST_ASSERT(!ioas_map_user(iommufd_fd, alloc.out_ioas_id, cq_buf, IOVA_CQ, 0x1000),
		    "IOAS_MAP(cq) errno=%d", errno);
    ...
}

Since iommufd requires mapping lengths and IOVAs to be aligned to the system
page size, won't this fail on kernels running with 16KB or 64KB page sizes?

[Severity: Medium]
Can a data race occur on weakly-ordered architectures without memory barriers?

The custom MMIO accessors use plain volatile writes:

tools/testing/selftests/kvm/gmem_provider_nvme_dma_test.c:wr32() {
    ...
static void     wr32(void *b, uint32_t o, uint32_t v) { *(volatile uint32_t *)((char *)b + o) = v; }
    ...
}

When ringing the doorbell:

tools/testing/selftests/kvm/gmem_provider_nvme_dma_test.c:main() {
    ...
	/* Ring SQ0 tail doorbell. */
	wr32(bar, NVME_SQ0TDBL, 1);
    ...
}

Without an explicit memory barrier, is the CPU free to reorder the MMIO write
to the doorbell ahead of the normal RAM writes to the Submission Queue Entry?

[Severity: Medium]
Does this code corrupt device state on big-endian architectures?

Since the test is now built for all architectures, it will run on big-endian
systems. However, the test uses native-endian assignments to interact with
little-endian PCI configuration space and NVMe registers:

tools/testing/selftests/kvm/gmem_provider_nvme_dma_test.c:main() {
    ...
	TEST_ASSERT(pread(vfio_fd, &pci_cmd, 2, cfg_off + 0x04) == 2, "cfg CMD read");
	pci_cmd |= 0x0006;
	TEST_ASSERT(pwrite(vfio_fd, &pci_cmd, 2, cfg_off + 0x04) == 2, "cfg CMD write");
    ...
}

And for the NVMe SQE:

tools/testing/selftests/kvm/gmem_provider_nvme_dma_test.c:main() {
    ...
		s[0]  = 0x00010006;	/* CID=1 << 16 | OPC=Identify */
    ...
}

Will these direct native-endian accesses produce byte-swapped data on
big-endian CPUs and cause the test to fail?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/f09647f427565c1789a6979470045aab78661563.1784194290.git.dwmw@amazon.co.uk?part=11