[RFC PATCH 00/14] guest_memfd: Export guest_memfd files to iommufd

[RFC PATCH 00/14] guest_memfd: Export guest_memfd files to iommufd

[Fred Griffoul]

From: Fred Griffoul <fgriffo@amazon.co.uk>

This series follows Ackerley's RFC v2, "guest_memfd alternative
providers" [1], and applies on top of it (kvm/next 7e60162b16f9 plus
his 50 patches). It answers one question his RFC leaves open: how the
IOMMU mappings of a device assigned to the guest follow guest_memfd.

A device cannot fault, and iommufd cannot pin guest_memfd pages: a
conversion to private, a punch hole, memory failure or a provider's
revoke must take a page away from the device too, and in PFN mode
there is no page to pin. So guest_memfd has to drive the device
mapping. This series proposes an interface for that, uses it in
iommufd as a demonstration, and adds the two PFN mode pieces it relies
on.

The interface
=============

A subsystem that maps guest_memfd pages, such as iommufd, uses three
functions and one callback, all in include/linux/guest_memfd.h:

  kvm_gmem_export_attach()      start following a guest_memfd file
  kvm_gmem_export_lookup_pfn()  which frame backs this page? A frame for
                                a shared page with memory, nothing for a
                                private page or a page without memory
  kvm_gmem_export_detach()      stop following it
  update(start, nr_pages)       the subsystem's callback: the answer for
                                this range may have changed, look it up
                                again and fix the mappings

No reference is taken on the frames. Instead, guest_memfd calls
update() on every change: conversions, punch holes, memory failure,
pages getting memory, and a PFN mode provider taking frames back or
giving them. The subsystem can rely on this:

  - the old frames stay valid until update() returns, so a device that
    never faults can switch each mapping in one step, to the new frame
    or to a scratch page; after it returns, the old frames are no
    longer used;
  - update() cannot fail (see the open questions);
  - update() may be called for pages that did not change;
  - it runs under guest_memfd's invalidate lock.

For the update to come before a frame is freed, a punch hole keeps the
frames until then: folios are held until update() returns, and in PFN
mode the range reads as holes until the update has run, before the
provider gets its frames back.

A PFN mode provider takes frames back with kvm_gmem_revoke(), from
Ackerley's RFC. When it gives frames to pages that had none, or makes a
read-only frame writable, it calls the new kvm_gmem_restore(), the
"counterpart to kvm_gmem_revoke()" that his sample leaves for users that
cannot fault (48/50). Nothing is taken away then, so restore only calls
update(); KVM and the VMM fault the pages in themselves.

Why not dma-buf: our RFC that backed guest_memfd with an imported
dma-buf [3] needed a dma-buf call for importers to build their own page
tables, and ranged invalidation, and Christian rejected both: importers
filling page tables is a no-go for dma-buf. A dma-buf also knows
nothing of shared and private pages, and folio mode guest_memfd needs
the same tracking as PFN mode. So the interface lives in guest_memfd.

iommufd, as a demonstration
===========================

IOMMU_IOAS_MAP_FILE accepts a guest_memfd. Its pages are not pinned:
each update() unmaps the range and maps what a lookup returns now, with
PAGE_SIZE IOPTEs, so a device that touches the range during an update
faults. Dirty tracking, the no-IOMMU mode and kernel access are
refused. This tests the interface and is not meant to be merged: real
support needs large IOPTEs cut and merged in place, IOPTEs replaced in
one step, and a scratch page for holes, all generic_pt work (Jason, on
our v2), and will be a separate iommufd series.

PFN mode pieces
===============

Two PFN mode changes come first, both raised in replies to Ackerley's
series:

  - get_pfn()'s order now means the naturally aligned block around the
    index, and KVM clamps it to a block it can map, aligned and of one
    attribute; today it can map a block over the wrong frames, or over
    shared and private pages. Invalidations of a PFN mode file cover
    the 1G-aligned blocks around their range, so that a fault that
    sized a block before the change retries.
  - get_pfn() gains an attributes output, with read only as its first
    bit, which the guest's, the VMM's and the IOMMU's mappings follow.

Open questions
==============

  - Private pages. The export gives shared pages only. That suits
    devices that are not trusted, and TDX, where a shared IOPTE on a
    private page is fatal. A trusted device (TDISP) needs private pages
    too: on AMD the IOMMU maps them, with IOPTE sizes that follow the
    RMP. The choice depends on the architecture and the device, so
    userspace would make it through iommufd, which would pass an attach
    flag asking lookups for private frames too, refused where the
    architecture cannot take it. Is an attach flag the right place for
    it?
  - update() cannot fail, but a consumer may need memory to apply one:
    iommufd allocates page tables to map a range again, and will need
    one to cut a large IOPTE. The demonstration only warns when that
    fails. Either the consumer reserves page tables when a range is
    mapped, or guest_memfd adds a prepare step before a conversion,
    which could fail and abort it. A provider's revoke cannot be
    aborted, so it would need the reservation either way. Which would
    you prefer?

Patches
=======

  1-2   PFN mode: clamp the order KVM maps, read-only frames.
  3-7   the export interface: folio mode, truncation, PFN mode, hole
        sizes, and updates when pages get memory.
  8-10  iommufd: track revocable domains, map guest_memfd files, and
        mock-domain IOVA queries for the test.
  11    kvm_gmem_restore().
  12-13 the sample provider: read-only pages, large blocks, and backing
        pages on demand and freeing them on a punch hole.
  14    gmem_conversion_iommufd_test.

Patch 2 also puts back the return after a failed kvm_gmem_get_pfn() in
arm64's gmem_abort(), which a kvm/next merge dropped; the fix alone
will be sent separately.

Testing
=======

On x86_64 in QEMU, with KASAN and lockdep:

  - gmem_conversion_iommufd_test maps the guest_memfd of a
    KVM_X86_SW_PROTECTED_VM into an iommufd mock domain, backed by
    folios, a tmpfs resource or the sample provider in PFN mode, and
    checks that the guest, the VMM and the device all see each change:
    conversions, allocations, punch holes, memory failure, and the
    provider's revokes and restores;
  - Ackerley's tests pass as on his RFC alone;
  - the iommufd selftests pass, except 12 dmabuf and
    access_domain_destory tests that also fail on his RFC alone here;
  - each patch builds with KVM and iommufd built in, with KVM as a
    module, with both as modules, and with SEV.

arm64 is neither built nor run.

History
=======

Our v2 [2] added an interface in mm for drivers that lend memory, with
guest_memfd and iommufd as two consumers of a provider file. Ackerley's
RFC now gives guest_memfd PFN mode providers itself and takes v2's
sample provider, so this series drops the mm interface: iommufd maps
the guest_memfd, not the provider file, and follows conversions.

[1] https://lore.kernel.org/all/20261022-gmem-alternative-providers-v2-0-5dd1dc6c660a@google.com/
[2] https://lore.kernel.org/all/20261014093733.90312-1-griffoul@gmail.com/
[3] https://lore.kernel.org/all/20261005095329.49764-1-griffoul@gmail.com/

prerequisite-message-id: <20261022-gmem-alternative-providers-v2-0-5dd1dc6c660a@google.com>

Fred Griffoul (14):
  KVM: guest_memfd: Clamp the order of PFN mode blocks to what KVM can
    map
  KVM: guest_memfd: Support read-only frames from PFN mode providers
  KVM: guest_memfd: Allow other subsystems to map guest_memfd files
  KVM: guest_memfd: Keep truncated folios until the exports are updated
  KVM: guest_memfd: Export PFN mode files
  KVM: guest_memfd: Report the size of holes in export lookups
  KVM: guest_memfd: Update exports when pages are allocated
  iommufd: Track the domains of revocable pages in struct iopt_pages
  iommufd: Map guest_memfd files
  iommufd/selftest: Add mock-domain IOVA queries
  KVM: guest_memfd: Add kvm_gmem_restore() for PFN mode providers
  samples/kvm: Add read-only pages and large blocks to the sample
    provider
  samples/kvm: Back absent pages on demand in the sample provider
  KVM: selftests: Test guest_memfd mapped by iommufd

 arch/arm64/kvm/mmu.c                          |  10 +-
 arch/arm64/kvm/nested.c                       |   9 +-
 arch/x86/kvm/Makefile                         |   4 +-
 arch/x86/kvm/mmu/mmu.c                        |  15 +-
 arch/x86/kvm/svm/sev.c                        |   4 +-
 drivers/iommu/iommufd/io_pagetable.c          |  38 +-
 drivers/iommu/iommufd/io_pagetable.h          |  61 +-
 drivers/iommu/iommufd/iommufd_test.h          |  18 +
 drivers/iommu/iommufd/pages.c                 | 366 ++++++-
 drivers/iommu/iommufd/selftest.c              | 125 +++
 include/linux/guest_memfd.h                   |  93 +-
 include/linux/kvm_host.h                      |   6 +-
 include/uapi/linux/iommufd.h                  |  11 +-
 samples/kvm/mem_provider_sample.c             | 118 ++-
 samples/kvm/mem_provider_sample.h             |  18 +-
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../kvm/x86/gmem_conversion_iommufd_test.c    | 956 ++++++++++++++++++
 virt/kvm/guest_memfd.c                        | 601 ++++++++++-
 18 files changed, 2329 insertions(+), 125 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/x86/gmem_conversion_iommufd_test.c


base-commit: 7e60162b16f9da0d2fa83da5c1723b6853a0ac1d
prerequisite-patch-id: d0932c65d2a453e84f7917da27f9f6ae1da95856
prerequisite-patch-id: 7be8f39fdcfe7e504a9ac85e835a5dfb2ee414ba
prerequisite-patch-id: c6d0df8d4fdd910000ed9e35f514db6ac32e6e11
prerequisite-patch-id: 45cd3951b114a26eac94390727287c1221862b5c
prerequisite-patch-id: 6d978b6f0561697e5b145e4b82d79ac8c89f9f31
prerequisite-patch-id: 9bd6979aa47dd00c87a3aa8feb1798486107ac4b
prerequisite-patch-id: 98438bd35d53340738a20fd8724b038c9de1207e
prerequisite-patch-id: 146dfaa64e29a4930adb60e9524c7613f888a241
prerequisite-patch-id: 170e27bc053fede8d24accca69809d3687faf668
prerequisite-patch-id: 789fe5bfb1183f6b9a9b0d817e5d1050a7999e1a
prerequisite-patch-id: 5dd6878d0476a9dd925aa8d7ef1cf18900cbcd82
prerequisite-patch-id: 1f0d1dca2a51df2283f1d83ed57dd9ce4e11b025
prerequisite-patch-id: 503eb8fb0bf9af163f344c2693d374b6a3093fd1
prerequisite-patch-id: 94e4777a58b357458bdce680bbb41c04c0c661f4
prerequisite-patch-id: 4c1f6b548994d239e5093ee1d268ef5a12a76873
prerequisite-patch-id: f6d13ac583d54fe1ae22adde87f5c68d8e8e41ce
prerequisite-patch-id: 7846b44d34507a9af6ac8a3fb263ee446ea56a1d
prerequisite-patch-id: 2d309198e27a3922b5cf5805ac5660ea92029219
prerequisite-patch-id: e1f75965298956ad87bb78969461eff31419b6eb
prerequisite-patch-id: 40ef669df2a64e5b28127059e4cea288d426f6f6
prerequisite-patch-id: d8adb8f100cb13423db02e7e7a834fa241ff04fe
prerequisite-patch-id: 75591ca138e7de26db6cd3e242a7f0ce56c4752c
prerequisite-patch-id: b1e2c46b473783f37490c29140498a0bb35ec7d3
prerequisite-patch-id: bc8b607986a132eef4daa5e20979d3f038d046ba
prerequisite-patch-id: c9de5cd8a95f8102597b49dcf70b8ec5825b9ec7
prerequisite-patch-id: 954606132449080c4e173eb06d114504f87d0322
prerequisite-patch-id: 82a44a46cf9d4cc3fcdc03c362fc4e86d1fc533a
prerequisite-patch-id: 23dbc9968c0b2570f40cd6c3b13337078334aa8d
prerequisite-patch-id: 6c6a5ca6d0a1b60158a47ed21f25889dd71dd9b7
prerequisite-patch-id: e4d1ba4119277d7772dbf6e81f748beab4e62856
prerequisite-patch-id: a32d69bdd1fb21b85547fdc026475e2d94ffbc28
prerequisite-patch-id: 2704fe13a294e1f33cfbee6cd27c11ee392a8b78
prerequisite-patch-id: ebb701b7e050327e798c172d897021806f7e6c35
prerequisite-patch-id: 03db7e48c976a019ce478883002670c4b1a33cd9
prerequisite-patch-id: a18608129e63bbbc6e89a4ca237863ba4c506a63
prerequisite-patch-id: a35607a73bd0329fd91a39bd7b9c855edb4b8798
prerequisite-patch-id: 6d57cd16d6c6e71c3da069c1d9eee1c86bdeee8d
prerequisite-patch-id: b64a5a718f540e7daf2dc4f0703422ad2cc52ebe
prerequisite-patch-id: eb9a9e4d7f4b7e439c5687786628f07c406864a4
prerequisite-patch-id: 2fe22fda60b4e4678bcf62d87bfd82181556af98
prerequisite-patch-id: 26211cbacdbc31a62524aa5ad402b0f3933d433e
prerequisite-patch-id: 02455f9ee70487d8b30b6aada5f5b62ff4e5ebe2
prerequisite-patch-id: 113da62f7845c2a9f86d24ed178cf52756917cc0
prerequisite-patch-id: 9ef3e8b12bc70690470035f0c6f351fa49562659
prerequisite-patch-id: 0953a92b86c0790554b64ebda5d6aed038a6cf0e
prerequisite-patch-id: 877e1c79e389936278122aa1efbdf99053b2d6c7
prerequisite-patch-id: 0b04c869d49a4c1ad96ff7b5d8e3b11c4069f23f
prerequisite-patch-id: 942b7480ddb281b9df74705f1a3a6e23a2440a43
prerequisite-patch-id: 98c3864233a7ba6d789ad260f4af170a020d2929
prerequisite-patch-id: 40dc0f2b1ab5e59f1f5de0be5134101ca896d13d
-- 
2.47.3

[RFC PATCH 01/14] KVM: guest_memfd: Clamp the order of PFN mode blocks to what KVM can map

[Fred Griffoul]

From: Fred Griffoul <fgriffo@amazon.co.uk>

KVM maps guest memory in 4K, 2M or 1G blocks, and uses guest_memfd's
max_order to choose. A folio of order N is physically aligned and never
mixes shared and private pages, so the order needs no check. A PFN mode
provider's frames need not be aligned and the provider does not know
which pages are private, yet kvm_gmem_get_pfn() passes its order to KVM
unchanged. KVM can then map a block over the wrong frames, or with one
attribute over shared and private pages.

Define the provider's order as that of the naturally aligned block
around the index, preset to 0. Clamp it to a block size KVM supports,
in which the gfn, the index and the pfn are aligned alike and every
page has the same attribute.

A fault that sized its block just before a revoke or conversion of
another page in the block would not retry, since KVM only retries
faults inside the invalidated range. Make every invalidation of a PFN
mode file cover the 1G-aligned blocks around its range.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Fred Griffoul <fgriffo@amazon.co.uk>
---
 include/linux/guest_memfd.h |  5 +--
 virt/kvm/guest_memfd.c      | 63 +++++++++++++++++++++++++++++++++++--
 2 files changed, 63 insertions(+), 5 deletions(-)

diff --git a/include/linux/guest_memfd.h b/include/linux/guest_memfd.h
index 3e2c0714ca33..a38a81693422 100644
--- a/include/linux/guest_memfd.h
+++ b/include/linux/guest_memfd.h
@@ -21,8 +21,9 @@ struct mempolicy;
  * @get_pfn: Obtain a PFN for a given index. The provider must ensure backing
  *           memory is present and is responsible for its contents, such as
  *           zeroing if necessary. On success, return 0 with the physical frame
- *           number in @pfn and, if @max_order is non-NULL, the maximum page
- *           order of contiguous backing memory containing @pfn. Returns a
+ *           number in @pfn and, if @max_order is non-NULL, the largest order
+ *           for which the naturally aligned block of 2^order pages that
+ *           contains @index is backed by contiguous frames. Returns a
  *           negative error code on failure. The provider must define
  *           @lookup_pfn_range when @get_pfn is defined.
  * @lookup_pfn_range: Look up the PFN for a given index. Returns 0 on success
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 3c4bedceccfc..be26bd16d420 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -147,6 +147,8 @@ static int gmem_provider_get_pfn(struct gmem_inode *gi, pgoff_t index,
 	unsigned long raw_pfn;
 	int r;
 
+	if (max_order)
+		*max_order = 0;
 	r = gi->provider_ops->get_pfn(gi->provider, index, &raw_pfn, max_order);
 	if (r)
 		return r;
@@ -326,10 +328,13 @@ static enum kvm_gfn_range_filter kvm_gmem_get_all_gfns_filter(struct inode *inod
 	return KVM_FILTER_PRIVATE;
 }
 
-static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start,
+static void __kvm_gmem_invalidate_start(struct inode *inode,
+					struct gmem_file *f, pgoff_t start,
 					pgoff_t end,
 					enum kvm_gfn_range_filter attr_filter)
 {
+	const gfn_t blk = kvm_gmem_is_pfn_mode(GMEM_I(inode)) ?
+			  1UL << PUD_ORDER : 1;
 	bool flush = false, found_memslot = false;
 	struct kvm_memory_slot *slot;
 	struct kvm *kvm = f->kvm;
@@ -355,6 +360,16 @@ static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start,
 
 		flush |= kvm_mmu_unmap_gfn_range(kvm, &gfn_range);
 
+		/*
+		 * In PFN mode, a fault may map a block of up to PUD_ORDER
+		 * pages, sized before this invalidation (see
+		 * kvm_gmem_pfn_map_order()): make faults anywhere in the
+		 * blocks around the range retry.
+		 */
+		kvm_mmu_invalidate_range_add(kvm,
+					     round_down(gfn_range.start, blk),
+					     round_up(gfn_range.end, blk));
+
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
 		kvm_arch_gmem_invalidate_range(kvm, &gfn_range);
 #endif
@@ -374,7 +389,7 @@ static void kvm_gmem_invalidate_start(struct inode *inode, pgoff_t start,
 	struct gmem_file *f;
 
 	kvm_gmem_for_each_file(f, inode)
-		__kvm_gmem_invalidate_start(f, start, end, filter);
+		__kvm_gmem_invalidate_start(inode, f, start, end, filter);
 }
 
 static void __kvm_gmem_invalidate_end(struct gmem_file *f, pgoff_t start,
@@ -670,7 +685,7 @@ static int kvm_gmem_release(struct inode *inode, struct file *file)
 	 * Zap all SPTEs pointed at by this file.  Do not free the backing
 	 * memory, as its lifetime is associated with the inode, not the file.
 	 */
-	__kvm_gmem_invalidate_start(f, 0, -1ul,
+	__kvm_gmem_invalidate_start(inode, f, 0, -1ul,
 				    kvm_gmem_get_all_gfns_filter(inode));
 	__kvm_gmem_invalidate_end(f, 0, -1ul);
 
@@ -1535,6 +1550,44 @@ static struct folio *__kvm_gmem_get_pfn(struct file *file,
 	return folio;
 }
 
+/*
+ * The order a PFN mode block can be mapped at: the provider's order, no
+ * larger than the alignment of @gfn and @pfn to @index allows, rounded down to
+ * a page-table size, and never over pages that are not all shared or all
+ * private, since KVM maps a block with a single attribute.  Called with the
+ * invalidate lock held; an invalidation of any page of the block makes the
+ * fault retry (see __kvm_gmem_invalidate_start()).
+ */
+static int kvm_gmem_pfn_map_order(struct inode *inode, pgoff_t index,
+				  gfn_t gfn, kvm_pfn_t pfn, int order)
+{
+	u64 misalign = (gfn ^ index) | (pfn ^ index);
+	u64 attrs;
+
+	if (WARN_ON_ONCE(order < 0))
+		return 0;
+
+	order = min(order, PUD_ORDER);
+	if (misalign)
+		order = min_t(int, order, __ffs64(misalign));
+
+	if (order >= PUD_ORDER)
+		order = PUD_ORDER;
+	else if (order >= PMD_ORDER)
+		order = PMD_ORDER;
+	else
+		return 0;
+
+	attrs = kvm_gmem_get_attributes(inode,
+					mtree_load(&GMEM_I(inode)->attributes, index));
+	while (order &&
+	       !kvm_gmem_range_has_attributes(inode,
+					      ALIGN_DOWN(index, 1UL << order),
+					      1UL << order, attrs))
+		order = order == PUD_ORDER ? PMD_ORDER : 0;
+	return order;
+}
+
 int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 		     gfn_t gfn, kvm_pfn_t *pfn, int *max_order)
 {
@@ -1556,6 +1609,10 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 		goto out;
 	}
 
+	if (kvm_gmem_is_pfn_mode(GMEM_I(file_inode(file))))
+		*max_order = kvm_gmem_pfn_map_order(file_inode(file), index, gfn,
+						    *pfn, *max_order);
+
 	if (kvm_arch_has_gmem_convert() &&
 	    kvm_gmem_is_private_mem(file_inode(file), index))
 		r = kvm_arch_gmem_make_private(kvm, gfn, *pfn,
-- 
2.47.3

[RFC PATCH 02/14] KVM: guest_memfd: Support read-only frames from PFN mode providers

[Fred Griffoul]

From: Fred Griffoul <fgriffo@amazon.co.uk>

A PFN mode provider may lend a frame that the guest must not write,
e.g. one shared by several guests, but guest_memfd maps every frame
writable.

Add an attributes output to get_pfn(), with GUEST_MEMFD_PFN_READONLY as
its first bit. KVM then maps such a frame read only and exits to
userspace on a guest write, and the VMM's mapping raises SIGBUS on a
write. A read-only frame cannot back a private page or be populated.

On arm64, the new exit in gmem_abort() also restores the return after a
failed kvm_gmem_get_pfn() that a kvm/next merge dropped.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Fred Griffoul <fgriffo@amazon.co.uk>
---
 arch/arm64/kvm/mmu.c              |  10 ++-
 arch/arm64/kvm/nested.c           |   9 ++-
 arch/x86/kvm/mmu/mmu.c            |  15 +++-
 arch/x86/kvm/svm/sev.c            |   4 +-
 include/linux/guest_memfd.h       |  24 +++++-
 include/linux/kvm_host.h          |   6 +-
 samples/kvm/mem_provider_sample.c |   2 +-
 virt/kvm/guest_memfd.c            | 128 ++++++++++++++++++++++++++----
 8 files changed, 168 insertions(+), 30 deletions(-)

diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index 745cb86f9e04..cdf6d671158a 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -1736,7 +1736,7 @@ static gfn_t get_canonical_gfn(const struct kvm_s2_fault_desc *s2fd,
 static int gmem_abort(const struct kvm_s2_fault_desc *s2fd,
 		      struct kvm_s2_fault_result *result)
 {
-	bool write_fault, exec_fault;
+	bool write_fault, exec_fault, writable;
 	bool perm_fault = kvm_s2_fault_is_perm(s2fd);
 	enum kvm_pgtable_walk_flags flags = KVM_PGTABLE_WALK_SHARED;
 	enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_R;
@@ -1773,17 +1773,19 @@ static int gmem_abort(const struct kvm_s2_fault_desc *s2fd,
 	/* Pairs with the smp_wmb() in kvm_mmu_invalidate_end(). */
 	smp_rmb();
 
-	ret = kvm_gmem_get_pfn(kvm, s2fd->memslot, canonical_gfn, &s2vi.pfn, NULL);
-	if (ret) {
+	ret = kvm_gmem_get_pfn(kvm, s2fd->memslot, canonical_gfn, &s2vi.pfn, NULL,
+			       &writable);
+	if (ret || (write_fault && !writable)) {
 		/* If result is non-NULL this is a synthetic fault. */
 		if (!result)
 			kvm_prepare_memory_fault_exit(s2fd->vcpu, gfn_to_gpa(canonical_gfn),
 						      s2vi.vma_pagesize, write_fault, exec_fault,
 						      false);
 		kfree(mapping);
+		return ret ?: -EFAULT;
 	}
 
-	if (!(s2fd->memslot->flags & KVM_MEM_READONLY))
+	if (!(s2fd->memslot->flags & KVM_MEM_READONLY) && writable)
 		prot |= KVM_PGTABLE_PROT_W;
 
 	if (s2fd->nested)
diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c
index f1c414f1c3fa..ec3bcdc230cf 100644
--- a/arch/arm64/kvm/nested.c
+++ b/arch/arm64/kvm/nested.c
@@ -1615,12 +1615,19 @@ static int kvm_translate_vncr(struct kvm_vcpu *vcpu, bool *is_gmem)
 			return -EFAULT;
 		}
 	} else {
-		ret = kvm_gmem_get_pfn(vcpu->kvm, memslot, gfn, &pfn, NULL);
+		ret = kvm_gmem_get_pfn(vcpu->kvm, memslot, gfn, &pfn, NULL,
+				       &writable);
 		if (ret) {
 			kvm_prepare_memory_fault_exit(vcpu, vt->wr.pa, PAGE_SIZE,
 					      write_fault, false, false);
 			return ret;
 		}
+		/* The CPU writes the VNCR page: a read-only frame cannot back it. */
+		if (!writable) {
+			kvm_prepare_memory_fault_exit(vcpu, vt->wr.pa, PAGE_SIZE,
+					      true, false, false);
+			return -EFAULT;
+		}
 
 		writable = !(memslot->flags & KVM_MEM_READONLY);
 	}
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 5a2cc6a17a9b..1c50955e80f9 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -4675,6 +4675,7 @@ static int kvm_mmu_faultin_pfn_gmem(struct kvm_vcpu *vcpu,
 				    struct kvm_page_fault *fault)
 {
 	int max_order, r;
+	bool writable;
 
 	if (!kvm_slot_has_gmem(fault->slot)) {
 		kvm_mmu_prepare_memory_fault_exit(vcpu, fault);
@@ -4682,13 +4683,23 @@ static int kvm_mmu_faultin_pfn_gmem(struct kvm_vcpu *vcpu,
 	}
 
 	r = kvm_gmem_get_pfn(vcpu->kvm, fault->slot, fault->gfn, &fault->pfn,
-			     &max_order);
+			     &max_order, &writable);
 	if (r) {
 		kvm_mmu_prepare_memory_fault_exit(vcpu, fault);
 		return r;
 	}
 
-	fault->map_writable = !(fault->slot->flags & KVM_MEM_READONLY);
+	/*
+	 * guest_memfd may report a read-only frame: map it without write
+	 * permission, and exit to userspace on a guest write rather than
+	 * install it.
+	 */
+	fault->map_writable = !(fault->slot->flags & KVM_MEM_READONLY) &&
+			      writable;
+	if (fault->write && !fault->map_writable) {
+		kvm_mmu_prepare_memory_fault_exit(vcpu, fault);
+		return -EFAULT;
+	}
 	fault->max_level = kvm_max_level_for_order(max_order);
 
 	return RET_PF_CONTINUE;
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index 6427db390b1c..4125b4fd30da 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -4096,7 +4096,7 @@ static void __sev_snp_reload_vmsa(struct kvm_vcpu *vcpu, gpa_t gpa)
 	 * The new VMSA will be private memory guest memory, so retrieve the
 	 * PFN from the gmem backend.
 	 */
-	if (kvm_gmem_get_pfn(vcpu->kvm, slot, gfn, &pfn, NULL))
+	if (kvm_gmem_get_pfn(vcpu->kvm, slot, gfn, &pfn, NULL, NULL))
 		return;
 
 	read_lock(&kvm->mmu_lock);
@@ -5092,7 +5092,7 @@ void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code)
 	mmu_seq = kvm->mmu_invalidate_seq;
 	smp_rmb();
 
-	ret = kvm_gmem_get_pfn(kvm, slot, gfn, &pfn, &order);
+	ret = kvm_gmem_get_pfn(kvm, slot, gfn, &pfn, &order, NULL);
 	if (ret) {
 		pr_warn_ratelimited("SEV: Unexpected RMP fault, no backing page for private GPA 0x%llx\n",
 				    gpa);
diff --git a/include/linux/guest_memfd.h b/include/linux/guest_memfd.h
index a38a81693422..9d8764b93af2 100644
--- a/include/linux/guest_memfd.h
+++ b/include/linux/guest_memfd.h
@@ -2,6 +2,7 @@
 #ifndef _LINUX_GUEST_MEMFD_H
 #define _LINUX_GUEST_MEMFD_H
 
+#include <linux/bits.h>
 #include <linux/types.h>
 
 struct file;
@@ -23,9 +24,12 @@ struct mempolicy;
  *           zeroing if necessary. On success, return 0 with the physical frame
  *           number in @pfn and, if @max_order is non-NULL, the largest order
  *           for which the naturally aligned block of 2^order pages that
- *           contains @index is backed by contiguous frames. Returns a
- *           negative error code on failure. The provider must define
- *           @lookup_pfn_range when @get_pfn is defined.
+ *           contains @index is backed by contiguous frames. @attrs, preset
+ *           to 0 by guest_memfd, receives the GUEST_MEMFD_PFN_* attributes
+ *           of the whole block. A provider that changes them must then
+ *           kvm_gmem_revoke() the range. Returns a negative error code on
+ *           failure. The provider must define @lookup_pfn_range when
+ *           @get_pfn is defined.
  * @lookup_pfn_range: Look up the PFN for a given index. Returns 0 on success
  *                    with the physical frame number in @pfn and the number of
  *                    contiguous pages the mapping is valid for in @nr_pages.
@@ -49,12 +53,24 @@ struct guest_memfd_provider_operations {
 				     struct mempolicy *mpol);
 	void (*invalidate_folio)(void *provider, struct folio *folio);
 	int (*get_pfn)(void *provider, pgoff_t index, unsigned long *pfn,
-		       int *max_order);
+		       int *max_order, u32 *attrs);
 	int (*lookup_pfn_range)(void *provider, pgoff_t index,
 				unsigned long *pfn, size_t *nr_pages);
 	void (*put_pfn_range)(void *provider, pgoff_t index, size_t nr_pages);
 };
 
+/*
+ * Attributes of a PFN mode block, reported by get_pfn().  Only the provider
+ * knows them, and every mapping of the block must follow them.
+ *
+ * READONLY: the guest may read the block but not write it, e.g. a page that
+ * the provider lends to several guests.  guest_memfd maps it read only for
+ * the guest and the VMM, and never backs a private page with it.
+ */
+#define GUEST_MEMFD_PFN_READONLY	BIT(0)
+
+#define GUEST_MEMFD_PFN_ATTRS		GUEST_MEMFD_PFN_READONLY
+
 #ifdef CONFIG_KVM_GUEST_MEMFD
 int kvm_gmem_revoke(struct inode *inode, pgoff_t start, size_t nr_pages,
 		    const unsigned long *pfn);
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 345f97b110fa..3d0db49ad08b 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -2628,11 +2628,13 @@ static inline u64 kvm_gmem_get_supported_flags(struct kvm *kvm)
 }
 
 int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
-		     gfn_t gfn, kvm_pfn_t *pfn, int *max_order);
+		     gfn_t gfn, kvm_pfn_t *pfn, int *max_order,
+		     bool *writable);
 #else
 static inline int kvm_gmem_get_pfn(struct kvm *kvm,
 				   struct kvm_memory_slot *slot, gfn_t gfn,
-				   kvm_pfn_t *pfn, int *max_order)
+				   kvm_pfn_t *pfn, int *max_order,
+				   bool *writable)
 {
 	KVM_BUG_ON(1, kvm);
 	return -EIO;
diff --git a/samples/kvm/mem_provider_sample.c b/samples/kvm/mem_provider_sample.c
index 5496e0c7864b..f30654a60dd8 100644
--- a/samples/kvm/mem_provider_sample.c
+++ b/samples/kvm/mem_provider_sample.c
@@ -179,7 +179,7 @@ static void mps_gmem_release(void *provider)
 }
 
 static int mps_gmem_get_pfn(void *provider, pgoff_t index, unsigned long *pfn,
-			    int *max_order)
+			    int *max_order, u32 *attrs)
 {
 	struct file *resource_file = provider;
 	struct mps_info *info = resource_file->private_data;
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index be26bd16d420..1554a17906da 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -142,18 +142,25 @@ static struct folio *gmem_provider_alloc_folio(struct gmem_inode *gi,
 }
 
 static int gmem_provider_get_pfn(struct gmem_inode *gi, pgoff_t index,
-				 kvm_pfn_t *pfn, int *max_order)
+				 kvm_pfn_t *pfn, int *max_order, u32 *attrs)
 {
 	unsigned long raw_pfn;
+	u32 __attrs = 0;
 	int r;
 
 	if (max_order)
 		*max_order = 0;
-	r = gi->provider_ops->get_pfn(gi->provider, index, &raw_pfn, max_order);
+	r = gi->provider_ops->get_pfn(gi->provider, index, &raw_pfn, max_order,
+				      &__attrs);
 	if (r)
 		return r;
 
+	if (WARN_ON_ONCE(__attrs & ~GUEST_MEMFD_PFN_ATTRS))
+		return -EIO;
+
 	*pfn = raw_pfn;
+	if (attrs)
+		*attrs = __attrs;
 	return 0;
 }
 
@@ -546,7 +553,7 @@ static long kvm_gmem_allocate(struct inode *inode, loff_t offset, loff_t len)
 				break;
 			}
 
-			r = gmem_provider_get_pfn(gi, index, &pfn, &max_order);
+			r = gmem_provider_get_pfn(gi, index, &pfn, &max_order, NULL);
 			if (r)
 				break;
 
@@ -721,13 +728,71 @@ static bool kvm_gmem_supports_mmap(struct inode *inode)
 	return GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_MMAP;
 }
 
+/* The protection for a host mapping of a PFN mode frame with @attrs. */
+static pgprot_t kvm_gmem_pfn_prot(struct vm_area_struct *vma, u32 attrs)
+{
+	vm_flags_t flags = vma->vm_flags;
+
+	if (attrs & GUEST_MEMFD_PFN_READONLY)
+		flags &= ~VM_WRITE;
+	return vm_get_page_prot(flags);
+}
+
+/*
+ * Map a PFN mode frame for the host.  Called with the invalidate lock held
+ * shared, so that a frame is never mapped after the revoke that removes it.
+ * A read-only frame is mapped without write permission, and a write to it
+ * raises SIGBUS.
+ */
+static vm_fault_t kvm_gmem_pfn_fault(struct vm_fault *vmf)
+{
+	struct inode *inode = file_inode(vmf->vma->vm_file);
+	struct vm_area_struct *vma = vmf->vma;
+	bool write = vmf->flags & FAULT_FLAG_WRITE;
+	kvm_pfn_t pfn;
+	u32 attrs;
+	int r;
+
+	r = gmem_provider_get_pfn(GMEM_I(inode), vmf->pgoff, &pfn, NULL, &attrs);
+	if (r)
+		return vmf_error(r);
+
+	if (write && (attrs & GUEST_MEMFD_PFN_READONLY))
+		return VM_FAULT_SIGBUS;
+
+	return vmf_insert_pfn_prot(vma, vmf->address, pfn,
+				   kvm_gmem_pfn_prot(vma, attrs));
+}
+
+/*
+ * A write to a read-only PTE, e.g. after mprotect(): allow it unless the frame
+ * is read only.  The core upgrades the PTE under its lock only if it did not
+ * change, and a provider that makes the frame read only revokes it first.
+ */
+static vm_fault_t kvm_gmem_pfn_mkwrite(struct vm_fault *vmf)
+{
+	struct inode *inode = file_inode(vmf->vma->vm_file);
+	vm_fault_t ret = 0;
+	kvm_pfn_t pfn;
+	u32 attrs;
+	int r;
+
+	filemap_invalidate_lock_shared(inode->i_mapping);
+	r = gmem_provider_get_pfn(GMEM_I(inode), vmf->pgoff, &pfn, NULL, &attrs);
+	if (r)
+		ret = vmf_error(r);
+	else if (attrs & GUEST_MEMFD_PFN_READONLY)
+		ret = VM_FAULT_SIGBUS;
+	filemap_invalidate_unlock_shared(inode->i_mapping);
+	return ret;
+}
+
 static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf)
 {
 	struct inode *inode = file_inode(vmf->vma->vm_file);
 	struct gmem_inode *gi = GMEM_I(inode);
 	vm_fault_t ret = VM_FAULT_LOCKED;
 	struct folio *folio = NULL;
-	kvm_pfn_t pfn;
 	int r = 0;
 
 	if (((loff_t)vmf->pgoff << PAGE_SHIFT) >= i_size_read(inode))
@@ -737,9 +802,7 @@ static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf)
 	if (!kvm_gmem_is_shared_mem(inode, vmf->pgoff)) {
 		r = -EACCES;
 	} else if (kvm_gmem_is_pfn_mode(gi)) {
-		r = gmem_provider_get_pfn(gi, vmf->pgoff, &pfn, NULL);
-		if (!r)
-			ret = vmf_insert_pfn(vmf->vma, vmf->address, pfn);
+		ret = kvm_gmem_pfn_fault(vmf);
 	} else {
 		folio = kvm_gmem_get_folio(inode, vmf->pgoff);
 		if (IS_ERR(folio))
@@ -815,6 +878,12 @@ static const struct vm_operations_struct kvm_gmem_vm_ops = {
 #endif
 };
 
+/* PFN mode: a write to a page mapped read only comes back to guest_memfd. */
+static const struct vm_operations_struct kvm_gmem_pfn_vm_ops = {
+	.fault		= kvm_gmem_fault_user_mapping,
+	.pfn_mkwrite	= kvm_gmem_pfn_mkwrite,
+};
+
 static int kvm_gmem_mmap(struct file *file, struct vm_area_struct *vma)
 {
 	struct gmem_inode *gi = GMEM_I(file_inode(file));
@@ -827,8 +896,11 @@ static int kvm_gmem_mmap(struct file *file, struct vm_area_struct *vma)
 		return -EINVAL;
 	}
 
-	if (kvm_gmem_is_pfn_mode(gi))
+	if (kvm_gmem_is_pfn_mode(gi)) {
 		vm_flags_set(vma, VM_PFNMAP);
+		vma->vm_ops = &kvm_gmem_pfn_vm_ops;
+		return 0;
+	}
 
 	vma->vm_ops = &kvm_gmem_vm_ops;
 
@@ -1503,7 +1575,7 @@ void kvm_gmem_unbind(struct kvm_memory_slot *slot)
 static struct folio *__kvm_gmem_get_pfn(struct file *file,
 					struct kvm_memory_slot *slot,
 					pgoff_t index, kvm_pfn_t *pfn,
-					int *max_order)
+					int *max_order, u32 *attrs)
 {
 	struct file *slot_file = READ_ONCE(slot->gmem.file);
 	struct gmem_inode *gi = GMEM_I(file_inode(file));
@@ -1522,7 +1594,7 @@ static struct folio *__kvm_gmem_get_pfn(struct file *file,
 	}
 
 	if (kvm_gmem_is_pfn_mode(gi)) {
-		r = gmem_provider_get_pfn(gi, index, pfn, max_order);
+		r = gmem_provider_get_pfn(gi, index, pfn, max_order, attrs);
 		if (r)
 			return ERR_PTR(r);
 		return NULL;
@@ -1588,14 +1660,22 @@ static int kvm_gmem_pfn_map_order(struct inode *inode, pgoff_t index,
 	return order;
 }
 
+/*
+ * @writable, if not NULL, is set to whether the guest may write the page.  It
+ * may be NULL only for private pages, which are always writable.
+ */
 int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
-		     gfn_t gfn, kvm_pfn_t *pfn, int *max_order)
+		     gfn_t gfn, kvm_pfn_t *pfn, int *max_order,
+		     bool *writable)
 {
 	pgoff_t index = kvm_gmem_get_index(slot, gfn);
 	struct folio *folio;
 	int r = 0, __order;
+	u32 attrs = 0;
 
 	max_order = max_order ?: &__order;
+	if (writable)
+		*writable = true;
 
 	CLASS(gmem_get_file, file)(slot);
 	if (!file)
@@ -1603,15 +1683,28 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 
 	filemap_invalidate_lock_shared(file_inode(file)->i_mapping);
 
-	folio = __kvm_gmem_get_pfn(file, slot, index, pfn, max_order);
+	folio = __kvm_gmem_get_pfn(file, slot, index, pfn, max_order, &attrs);
 	if (IS_ERR(folio)) {
 		r = PTR_ERR(folio);
 		goto out;
 	}
 
-	if (kvm_gmem_is_pfn_mode(GMEM_I(file_inode(file))))
+	if (kvm_gmem_is_pfn_mode(GMEM_I(file_inode(file)))) {
+		/*
+		 * A read-only frame is not the guest's alone, so it never
+		 * backs a private page: a private fault on it is a hole.
+		 */
+		if (attrs & GUEST_MEMFD_PFN_READONLY) {
+			if (kvm_gmem_is_private_mem(file_inode(file), index) ||
+			    WARN_ON_ONCE(!writable)) {
+				r = -EFAULT;
+				goto out;
+			}
+			*writable = false;
+		}
 		*max_order = kvm_gmem_pfn_map_order(file_inode(file), index, gfn,
 						    *pfn, *max_order);
+	}
 
 	if (kvm_arch_has_gmem_convert() &&
 	    kvm_gmem_is_private_mem(file_inode(file), index))
@@ -1637,12 +1730,13 @@ static long __kvm_gmem_populate(struct kvm *kvm, struct kvm_memory_slot *slot,
 {
 	pgoff_t index = kvm_gmem_get_index(slot, gfn);
 	struct folio *folio;
+	u32 attrs = 0;
 	kvm_pfn_t pfn;
 	int ret;
 
 	filemap_invalidate_lock(file->f_mapping);
 
-	folio = __kvm_gmem_get_pfn(file, slot, index, &pfn, NULL);
+	folio = __kvm_gmem_get_pfn(file, slot, index, &pfn, NULL, &attrs);
 	if (IS_ERR(folio)) {
 		ret = PTR_ERR(folio);
 		goto out_unlock;
@@ -1651,6 +1745,12 @@ static long __kvm_gmem_populate(struct kvm *kvm, struct kvm_memory_slot *slot,
 	if (folio)
 		folio_unlock(folio);
 
+	/* Populated pages are private, and are written: not a read-only frame. */
+	if (attrs & GUEST_MEMFD_PFN_READONLY) {
+		ret = -EPERM;
+		goto out_put_folio;
+	}
+
 	if (!kvm_is_private_gfn(kvm, gfn)) {
 		ret = -EINVAL;
 		goto out_put_folio;
-- 
2.47.3

[RFC PATCH 03/14] KVM: guest_memfd: Allow other subsystems to map guest_memfd files

[Fred Griffoul]

From: Fred Griffoul <fgriffo@amazon.co.uk>

A device assigned to a guest must map the guest's shared memory and
lose a page whenever guest_memfd takes it away. iommufd cannot do that:
it pins what it maps, which guest_memfd does not allow, and a device
cannot fault.

Add an export interface. A subsystem attaches to a guest_memfd file,
looks up the frame behind each shared page without taking a reference,
and gets an update() callback when what a lookup returns for a range
may have changed. The old frames stay valid until update() returns, so
a device that never faults can switch each mapping in one step.

update() is called on conversions and memory failure; punch holes, PFN
mode and pages that get memory follow. The functions are exported for
symbol_get(), since KVM may be a module while the consumer is built in.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Fred Griffoul <fgriffo@amazon.co.uk>
---
 arch/x86/kvm/Makefile       |   3 +-
 include/linux/guest_memfd.h |  55 +++++++++++++
 virt/kvm/guest_memfd.c      | 149 ++++++++++++++++++++++++++++++++++++
 3 files changed, 206 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kvm/Makefile b/arch/x86/kvm/Makefile
index 5fd01bd19d1c..25fb3d6e2032 100644
--- a/arch/x86/kvm/Makefile
+++ b/arch/x86/kvm/Makefile
@@ -62,7 +62,8 @@ exports_grep_trailer := --include='*.[ch]' -nrw $(srctree)/virt/kvm $(srctree)/a
 				-e kvm_write_track_add_gfn \
 				-e kvm_write_track_remove_gfn \
 				-e kvm_file_to_kvm_fn \
-				-e kvm_gmem_revoke
+				-e kvm_gmem_revoke \
+				-e kvm_gmem_export
 
 # Force grep to emit a goofy group separator that can in turn be replaced with
 # the above newline macro (newlines in Make are a nightmare).  Note, grep only
diff --git a/include/linux/guest_memfd.h b/include/linux/guest_memfd.h
index 9d8764b93af2..fdf7f8404d6c 100644
--- a/include/linux/guest_memfd.h
+++ b/include/linux/guest_memfd.h
@@ -3,6 +3,7 @@
 #define _LINUX_GUEST_MEMFD_H
 
 #include <linux/bits.h>
+#include <linux/list.h>
 #include <linux/types.h>
 
 struct file;
@@ -82,4 +83,58 @@ static inline int kvm_gmem_revoke(struct inode *inode, pgoff_t start, size_t nr_
 }
 #endif
 
+/*
+ * Mapping guest_memfd files in other subsystems.
+ *
+ * A subsystem that maps guest_memfd pages, such as iommufd, can neither pin
+ * nor fault them.  It attaches to the file, looks up the frame behind each
+ * shared page without taking a reference, and guest_memfd calls its update()
+ * whenever what a lookup returns for a range may have changed: conversions and
+ * memory failure.  update() makes the consumer's mappings of the range match
+ * what a lookup returns now: a frame, or a hole for a private page or a page
+ * without a frame.  It may be called for pages that did not change, and for
+ * ranges that do not follow the blocks that lookups report: the consumer may
+ * then remap whole blocks, also outside the range, e.g. to merge them into
+ * larger mappings.  A consumer that can fault may instead only remove its
+ * mappings of the range.
+ *
+ * Until update() returns, the frames it replaces stay valid, so a consumer
+ * may switch each mapping to its new target in one step; once it returns,
+ * the consumer no longer accesses them.  update() cannot fail: a consumer
+ * that needs memory to apply one, e.g. to split a large mapping, reserves it
+ * beforehand.
+ *
+ * guest_memfd calls update() with its invalidate lock held, exclusive when
+ * pages convert and shared on memory failure, possibly with a folio of the
+ * file locked; so update() may only call kvm_gmem_export_lookup_pfn().  Under
+ * the shared lock, updates may run at the same time, also for overlapping
+ * ranges.  The consumer holds a lock across a lookup and the mapping it
+ * installs, which update() also takes.  Lock order:
+ *
+ *   guest_memfd invalidate lock
+ *     consumer lock
+ *
+ * A consumer attaches and detaches outside its lock.  KVM may be a module
+ * while the consumer is built in: consumers reach these with symbol_get().
+ */
+struct guest_memfd_export;
+
+struct guest_memfd_export_ops {
+	void (*update)(struct guest_memfd_export *e, pgoff_t start,
+		       pgoff_t nr_pages);
+};
+
+struct guest_memfd_export {
+	const struct guest_memfd_export_ops *ops;
+
+	/* Private to guest_memfd. */
+	struct list_head node;
+	struct file *file;
+};
+
+int kvm_gmem_export_attach(struct file *file, struct guest_memfd_export *e);
+void kvm_gmem_export_detach(struct guest_memfd_export *e);
+int kvm_gmem_export_lookup_pfn(struct guest_memfd_export *e, pgoff_t index,
+			       unsigned long *pfn, int *max_order, u32 *attrs);
+
 #endif /* _LINUX_GUEST_MEMFD_H */
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 1554a17906da..81031483a1f1 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -38,6 +38,8 @@ struct gmem_inode {
 
 	void *provider;
 	const struct guest_memfd_provider_operations *provider_ops;
+	/* Subsystems that map the file, see include/linux/guest_memfd.h. */
+	struct list_head exports;
 
 	u64 flags;
 	/*
@@ -420,6 +422,21 @@ static void kvm_gmem_invalidate_end(struct inode *inode, pgoff_t start,
 		__kvm_gmem_invalidate_end(f, start, end);
 }
 
+/*
+ * What a lookup returns for [start, end) may have changed: exclusive lock when
+ * pages converted, shared on memory failure.
+ */
+static void kvm_gmem_update_exports(struct inode *inode, pgoff_t start,
+				    pgoff_t end)
+{
+	struct guest_memfd_export *e;
+
+	lockdep_assert_held(&inode->i_mapping->invalidate_lock);
+
+	list_for_each_entry(e, &GMEM_I(inode)->exports, node)
+		e->ops->update(e, start, end - start);
+}
+
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
 static void kvm_gmem_pfn_reclaim(struct gmem_inode *gi, pgoff_t start, pgoff_t end)
 {
@@ -1130,6 +1147,9 @@ static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 
 	mas_store_prealloc(&mas, xa_mk_value(attrs));
 
+	/* After the store, before a vCPU can fault a private page in. */
+	kvm_gmem_update_exports(inode, start, end);
+
 	kvm_gmem_invalidate_end(inode, start, end);
 out:
 	filemap_invalidate_unlock(mapping);
@@ -1204,6 +1224,131 @@ static struct file_operations kvm_gmem_fops = {
 	.unlocked_ioctl	= kvm_gmem_ioctl,
 };
 
+/**
+ * kvm_gmem_export_attach - Map a guest_memfd file in another subsystem.
+ * @file: The guest_memfd file.
+ * @e:    The export, with @e->ops set.
+ *
+ * guest_memfd calls @e->ops->update() from now on, and holds a reference to
+ * @file until kvm_gmem_export_detach().  May sleep.
+ *
+ * Return: 0, -ENODEV if @file is not a guest_memfd file, or -EOPNOTSUPP in
+ * PFN mode.
+ */
+int kvm_gmem_export_attach(struct file *file, struct guest_memfd_export *e)
+{
+	struct inode *inode = file_inode(file);
+
+	if (file->f_op != &kvm_gmem_fops)
+		return -ENODEV;
+	if (kvm_gmem_is_pfn_mode(GMEM_I(inode)))
+		return -EOPNOTSUPP;
+
+	e->file = get_file(file);
+	filemap_invalidate_lock(inode->i_mapping);
+	list_add_tail(&e->node, &GMEM_I(inode)->exports);
+	filemap_invalidate_unlock(inode->i_mapping);
+	return 0;
+}
+EXPORT_SYMBOL_GPL(kvm_gmem_export_attach);
+
+/**
+ * kvm_gmem_export_detach - Stop mapping a guest_memfd file.
+ * @e: The export.
+ *
+ * Waits for an update() in progress.  May sleep.
+ */
+void kvm_gmem_export_detach(struct guest_memfd_export *e)
+{
+	struct inode *inode = file_inode(e->file);
+
+	filemap_invalidate_lock(inode->i_mapping);
+	list_del(&e->node);
+	filemap_invalidate_unlock(inode->i_mapping);
+	fput(e->file);
+}
+EXPORT_SYMBOL_GPL(kvm_gmem_export_detach);
+
+/* Are [index, index + nr) all shared?  Without the invalidate lock. */
+static bool kvm_gmem_range_is_shared(struct inode *inode, pgoff_t index,
+				     unsigned long nr)
+{
+	unsigned long first = index;
+	void *entry;
+
+	mt_for_each(&GMEM_I(inode)->attributes, entry, first, index + nr - 1) {
+		if (kvm_gmem_get_attributes(inode, entry) &
+		    KVM_MEMORY_ATTRIBUTE_PRIVATE)
+			return false;
+	}
+	return true;
+}
+
+/* What kvm_gmem_export_lookup_pfn() returns for a page inside the file. */
+static int __kvm_gmem_export_lookup(struct inode *inode, pgoff_t index,
+				    unsigned long *pfn, int *max_order,
+				    u32 *attrs)
+{
+	struct folio *folio;
+
+	if (!kvm_gmem_range_is_shared(inode, index, 1))
+		return -ENOENT;
+
+	/* A folio that was never zeroed is a hole. */
+	folio = filemap_get_folio(inode->i_mapping, index);
+	if (IS_ERR(folio))
+		return -ENOENT;
+	if (folio_test_hwpoison(folio) || !folio_test_uptodate(folio)) {
+		folio_put(folio);
+		return -ENOENT;
+	}
+	*pfn = folio_file_pfn(folio, index);
+	*max_order = 0;
+	*attrs = 0;
+	folio_put(folio);
+
+	/* Nor may the block hold a private page. */
+	while (*max_order &&
+	       !kvm_gmem_range_is_shared(inode,
+					 round_down(index, 1UL << *max_order),
+					 1UL << *max_order))
+		(*max_order)--;
+	return 0;
+}
+
+/**
+ * kvm_gmem_export_lookup_pfn - The frame behind a shared page.
+ * @e:         The export.
+ * @index:     The page index in the file.
+ * @pfn:       The frame.
+ * @max_order: In: the largest order the caller can map.  Out: the order of
+ *             the naturally aligned block around @index that is contiguous,
+ *             shared and of the same @attrs; 0 for -ENOENT.
+ * @attrs:     The GUEST_MEMFD_PFN_* attributes of the block.
+ *
+ * Takes no reference and allocates nothing, and may be called from the
+ * export's update(): the answer holds until the update() that changes it.
+ *
+ * Context: May sleep.
+ * Return: 0, -ENOENT for a private page or a page without a frame, -EINVAL
+ * beyond the end of the file, or another error from the provider.
+ */
+int kvm_gmem_export_lookup_pfn(struct guest_memfd_export *e, pgoff_t index,
+			       unsigned long *pfn, int *max_order, u32 *attrs)
+{
+	struct inode *inode = file_inode(e->file);
+	int r;
+
+	if (index >= i_size_read(inode) >> PAGE_SHIFT)
+		return -EINVAL;
+
+	r = __kvm_gmem_export_lookup(inode, index, pfn, max_order, attrs);
+	if (r == -ENOENT)
+		*max_order = 0;
+	return r;
+}
+EXPORT_SYMBOL_GPL(kvm_gmem_export_lookup_pfn);
+
 static int kvm_gmem_migrate_folio(struct address_space *mapping,
 				  struct folio *dst, struct folio *src,
 				  enum migrate_mode mode)
@@ -1241,6 +1386,9 @@ static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol
 	filter = kvm_gmem_get_all_gfns_filter(inode);
 	kvm_gmem_invalidate_start(inode, start, end, filter);
 
+	/* Lookups see the poisoned folio as a hole: take it from the exports. */
+	kvm_gmem_update_exports(inode, start, end);
+
 	/*
 	 * Do not truncate the range, what action is taken in response to the
 	 * error is userspace's decision (assuming the architecture supports
@@ -1867,6 +2015,7 @@ static struct inode *kvm_gmem_alloc_inode(struct super_block *sb)
 	gi->provider = NULL;
 	gi->provider_ops = NULL;
 	INIT_LIST_HEAD(&gi->gmem_file_list);
+	INIT_LIST_HEAD(&gi->exports);
 	return &gi->vfs_inode;
 }
 
-- 
2.47.3

[RFC PATCH 04/14] KVM: guest_memfd: Keep truncated folios until the exports are updated

[Fred Griffoul]

From: Fred Griffoul <fgriffo@amazon.co.uk>

A punch hole frees the folios it truncates at once, but an export may
still map them, since it holds no reference: a device would then write
into freed memory.

Truncate a folio mode range batch by batch, holding a reference on each
folio until the exports are updated, so that a folio goes back to the
allocator only once no device maps it.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Fred Griffoul <fgriffo@amazon.co.uk>
---
 include/linux/guest_memfd.h | 28 ++++++++++++++--------------
 virt/kvm/guest_memfd.c      | 30 ++++++++++++++++++++++++++++--
 2 files changed, 42 insertions(+), 16 deletions(-)

diff --git a/include/linux/guest_memfd.h b/include/linux/guest_memfd.h
index fdf7f8404d6c..ba3eebb1ed1a 100644
--- a/include/linux/guest_memfd.h
+++ b/include/linux/guest_memfd.h
@@ -89,14 +89,14 @@ static inline int kvm_gmem_revoke(struct inode *inode, pgoff_t start, size_t nr_
  * A subsystem that maps guest_memfd pages, such as iommufd, can neither pin
  * nor fault them.  It attaches to the file, looks up the frame behind each
  * shared page without taking a reference, and guest_memfd calls its update()
- * whenever what a lookup returns for a range may have changed: conversions and
- * memory failure.  update() makes the consumer's mappings of the range match
- * what a lookup returns now: a frame, or a hole for a private page or a page
- * without a frame.  It may be called for pages that did not change, and for
- * ranges that do not follow the blocks that lookups report: the consumer may
- * then remap whole blocks, also outside the range, e.g. to merge them into
- * larger mappings.  A consumer that can fault may instead only remove its
- * mappings of the range.
+ * whenever what a lookup returns for a range may have changed: conversions,
+ * punch holes and memory failure.  update() makes the consumer's mappings of
+ * the range match what a lookup returns now: a frame, or a hole for a private
+ * page or a page without a frame.  It may be called for pages that did not
+ * change, and for ranges that do not follow the blocks that lookups report:
+ * the consumer may then remap whole blocks, also outside the range, e.g. to
+ * merge them into larger mappings.  A consumer that can fault may instead only
+ * remove its mappings of the range.
  *
  * Until update() returns, the frames it replaces stay valid, so a consumer
  * may switch each mapping to its new target in one step; once it returns,
@@ -104,12 +104,12 @@ static inline int kvm_gmem_revoke(struct inode *inode, pgoff_t start, size_t nr_
  * that needs memory to apply one, e.g. to split a large mapping, reserves it
  * beforehand.
  *
- * guest_memfd calls update() with its invalidate lock held, exclusive when
- * pages convert and shared on memory failure, possibly with a folio of the
- * file locked; so update() may only call kvm_gmem_export_lookup_pfn().  Under
- * the shared lock, updates may run at the same time, also for overlapping
- * ranges.  The consumer holds a lock across a lookup and the mapping it
- * installs, which update() also takes.  Lock order:
+ * guest_memfd calls update() with its invalidate lock held, exclusive when it
+ * takes frames away or pages convert and shared on memory failure, possibly
+ * with a folio of the file locked; so update() may only call
+ * kvm_gmem_export_lookup_pfn().  Under the shared lock, updates may run at the
+ * same time, also for overlapping ranges.  The consumer holds a lock across a
+ * lookup and the mapping it installs, which update() also takes.  Lock order:
  *
  *   guest_memfd invalidate lock
  *     consumer lock
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 81031483a1f1..65a951fa49c2 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -424,7 +424,7 @@ static void kvm_gmem_invalidate_end(struct inode *inode, pgoff_t start,
 
 /*
  * What a lookup returns for [start, end) may have changed: exclusive lock when
- * pages converted, shared on memory failure.
+ * frames were taken away or pages converted, at least shared on memory failure.
  */
 static void kvm_gmem_update_exports(struct inode *inode, pgoff_t start,
 				    pgoff_t end)
@@ -437,6 +437,32 @@ static void kvm_gmem_update_exports(struct inode *inode, pgoff_t start,
 		e->ops->update(e, start, end - start);
 }
 
+/*
+ * Truncate [start, end) of a folio mode file.  The exports map the folios
+ * without references, so hold the folios until the exports are updated.
+ */
+static void kvm_gmem_truncate(struct inode *inode, pgoff_t start, pgoff_t end)
+{
+	struct folio_batch fbatch;
+	pgoff_t next = start;
+	unsigned int i;
+
+	folio_batch_init(&fbatch);
+	while (filemap_get_folios(inode->i_mapping, &next, end - 1, &fbatch)) {
+		pgoff_t first = max(fbatch.folios[0]->index, start);
+		pgoff_t last = min(next, end);
+
+		truncate_inode_pages_range(inode->i_mapping,
+					   (loff_t)first << PAGE_SHIFT,
+					   ((loff_t)last << PAGE_SHIFT) - 1);
+		kvm_gmem_update_exports(inode, first, last);
+		for (i = 0; i < folio_batch_count(&fbatch); i++)
+			folio_put(fbatch.folios[i]);
+		folio_batch_reinit(&fbatch);
+		cond_resched();
+	}
+}
+
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
 static void kvm_gmem_pfn_reclaim(struct gmem_inode *gi, pgoff_t start, pgoff_t end)
 {
@@ -499,7 +525,7 @@ static int __kvm_gmem_revoke(struct inode *inode, pgoff_t start,
 		if (put_pfn_range)
 			gmem_provider_put_pfn_range(GMEM_I(inode), start, nr_pages);
 	} else {
-		truncate_inode_pages_range(inode->i_mapping, offset, offset + len - 1);
+		kvm_gmem_truncate(inode, start, end);
 	}
 
 	kvm_gmem_invalidate_end(inode, start, end);
-- 
2.47.3

[RFC PATCH 05/14] KVM: guest_memfd: Export PFN mode files

[Fred Griffoul]

From: Fred Griffoul <fgriffo@amazon.co.uk>

PFN mode files cannot be attached yet, though a provider's frames are
the memory that most needs a device mapping that follows its revokes.

Allow them. A lookup asks the provider with lookup_pfn_range(), which
allocates nothing, then get_pfn() for the block and its attributes. A
revoke updates the exports before the frames are reclaimed. A punch hole
must remove the frames from the exports before the provider gets them
back, while the provider still backs them: lookups report the range as
holes until then, and the exports are updated again afterwards.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Fred Griffoul <fgriffo@amazon.co.uk>
---
 include/linux/guest_memfd.h |  20 ++++---
 virt/kvm/guest_memfd.c      | 111 ++++++++++++++++++++++++++++++------
 2 files changed, 105 insertions(+), 26 deletions(-)

diff --git a/include/linux/guest_memfd.h b/include/linux/guest_memfd.h
index ba3eebb1ed1a..937693ae9573 100644
--- a/include/linux/guest_memfd.h
+++ b/include/linux/guest_memfd.h
@@ -90,13 +90,13 @@ static inline int kvm_gmem_revoke(struct inode *inode, pgoff_t start, size_t nr_
  * nor fault them.  It attaches to the file, looks up the frame behind each
  * shared page without taking a reference, and guest_memfd calls its update()
  * whenever what a lookup returns for a range may have changed: conversions,
- * punch holes and memory failure.  update() makes the consumer's mappings of
- * the range match what a lookup returns now: a frame, or a hole for a private
- * page or a page without a frame.  It may be called for pages that did not
- * change, and for ranges that do not follow the blocks that lookups report:
- * the consumer may then remap whole blocks, also outside the range, e.g. to
- * merge them into larger mappings.  A consumer that can fault may instead only
- * remove its mappings of the range.
+ * punch holes, memory failure and a provider's kvm_gmem_revoke().  update()
+ * makes the consumer's mappings of the range match what a lookup returns now:
+ * a frame, or a hole for a private page or a page without a frame.  It may be
+ * called for pages that did not change, and for ranges that do not follow the
+ * blocks that lookups report: the consumer may then remap whole blocks, also
+ * outside the range, e.g. to merge them into larger mappings.  A consumer that
+ * can fault may instead only remove its mappings of the range.
  *
  * Until update() returns, the frames it replaces stay valid, so a consumer
  * may switch each mapping to its new target in one step; once it returns,
@@ -111,8 +111,10 @@ static inline int kvm_gmem_revoke(struct inode *inode, pgoff_t start, size_t nr_
  * same time, also for overlapping ranges.  The consumer holds a lock across a
  * lookup and the mapping it installs, which update() also takes.  Lock order:
  *
- *   guest_memfd invalidate lock
- *     consumer lock
+ *   PFN mode provider lock (across kvm_gmem_revoke())
+ *     guest_memfd invalidate lock
+ *       consumer lock
+ *         provider leaf lock (get_pfn(), lookup_pfn_range())
  *
  * A consumer attaches and detaches outside its lock.  KVM may be a module
  * while the consumer is built in: consumers reach these with symbol_get().
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 65a951fa49c2..015f6657c386 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -40,6 +40,12 @@ struct gmem_inode {
 	const struct guest_memfd_provider_operations *provider_ops;
 	/* Subsystems that map the file, see include/linux/guest_memfd.h. */
 	struct list_head exports;
+	/*
+	 * A PFN mode punch hole in progress: lookups find holes in
+	 * [punch_start, punch_end) until the provider has the frames back.
+	 * Written under the invalidate lock, read without it.
+	 */
+	pgoff_t punch_start, punch_end;
 
 	u64 flags;
 	/*
@@ -515,15 +521,32 @@ static int __kvm_gmem_revoke(struct inode *inode, pgoff_t start,
 	kvm_gmem_invalidate_start(inode, start, end, filter);
 
 	if (kvm_gmem_is_pfn_mode(GMEM_I(inode))) {
+		struct gmem_inode *gi = GMEM_I(inode);
+
 		unmap_mapping_range(inode->i_mapping, offset, len, 0);
+		/*
+		 * A punch hole gives the frames back only after the reclaim:
+		 * until then, lookups find holes.  Afterwards, the exports map
+		 * what the provider backs the range with.
+		 */
+		if (put_pfn_range) {
+			WRITE_ONCE(gi->punch_start, start);
+			/* Pairs with the smp_rmb() in kvm_gmem_export_pfn(). */
+			smp_wmb();
+			WRITE_ONCE(gi->punch_end, end);
+		}
+		kvm_gmem_update_exports(inode, start, end);
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
 		if (pfn)
 			kvm_arch_gmem_reclaim(*pfn, nr_pages);
 		else
-			kvm_gmem_pfn_reclaim(GMEM_I(inode), start, end);
+			kvm_gmem_pfn_reclaim(gi, start, end);
 #endif
-		if (put_pfn_range)
-			gmem_provider_put_pfn_range(GMEM_I(inode), start, nr_pages);
+		if (put_pfn_range) {
+			gmem_provider_put_pfn_range(gi, start, nr_pages);
+			WRITE_ONCE(gi->punch_end, 0);
+			kvm_gmem_update_exports(inode, start, end);
+		}
 	} else {
 		kvm_gmem_truncate(inode, start, end);
 	}
@@ -1258,8 +1281,7 @@ static struct file_operations kvm_gmem_fops = {
  * guest_memfd calls @e->ops->update() from now on, and holds a reference to
  * @file until kvm_gmem_export_detach().  May sleep.
  *
- * Return: 0, -ENODEV if @file is not a guest_memfd file, or -EOPNOTSUPP in
- * PFN mode.
+ * Return: 0, or -ENODEV if @file is not a guest_memfd file.
  */
 int kvm_gmem_export_attach(struct file *file, struct guest_memfd_export *e)
 {
@@ -1267,8 +1289,6 @@ int kvm_gmem_export_attach(struct file *file, struct guest_memfd_export *e)
 
 	if (file->f_op != &kvm_gmem_fops)
 		return -ENODEV;
-	if (kvm_gmem_is_pfn_mode(GMEM_I(inode)))
-		return -EOPNOTSUPP;
 
 	e->file = get_file(file);
 	filemap_invalidate_lock(inode->i_mapping);
@@ -1310,28 +1330,83 @@ static bool kvm_gmem_range_is_shared(struct inode *inode, pgoff_t index,
 	return true;
 }
 
+/*
+ * PFN mode: lookup_pfn_range() first, so that nothing is allocated, then
+ * get_pfn() for the order and attributes.
+ */
+static int kvm_gmem_export_pfn(struct gmem_inode *gi, pgoff_t index,
+			       unsigned long *pfn, int *max_order, u32 *attrs)
+{
+	pgoff_t punch_start, punch_end;
+	int order = *max_order, block;
+	unsigned long raw_pfn;
+	size_t nr_pages;
+	kvm_pfn_t frame;
+	int r;
+
+	/*
+	 * The end first: a new end comes with its start, and an old end of 0
+	 * makes the window empty.
+	 */
+	punch_end = READ_ONCE(gi->punch_end);
+	/* Pairs with the smp_wmb() in __kvm_gmem_revoke(). */
+	smp_rmb();
+	punch_start = READ_ONCE(gi->punch_start);
+	if (index >= punch_start && index < punch_end)
+		return -ENOENT;
+
+	r = gi->provider_ops->lookup_pfn_range(gi->provider, index, &raw_pfn,
+					       &nr_pages);
+	if (r)
+		return r;
+
+	r = gmem_provider_get_pfn(gi, index, &frame, &block, attrs);
+	if (r)
+		return r == -EFAULT ? -ENOENT : r;
+
+	order = min(order, block);
+	if (frame ^ index)
+		order = min_t(int, order, __ffs64(frame ^ index));
+	/* Nor may the block reach into a punch hole. */
+	while (order &&
+	       round_down(index, 1UL << order) < punch_end &&
+	       round_up(index + 1, 1UL << order) > punch_start)
+		order--;
+	*pfn = frame;
+	*max_order = order;
+	return 0;
+}
+
 /* What kvm_gmem_export_lookup_pfn() returns for a page inside the file. */
 static int __kvm_gmem_export_lookup(struct inode *inode, pgoff_t index,
 				    unsigned long *pfn, int *max_order,
 				    u32 *attrs)
 {
+	struct gmem_inode *gi = GMEM_I(inode);
 	struct folio *folio;
+	int r;
 
 	if (!kvm_gmem_range_is_shared(inode, index, 1))
 		return -ENOENT;
 
-	/* A folio that was never zeroed is a hole. */
-	folio = filemap_get_folio(inode->i_mapping, index);
-	if (IS_ERR(folio))
-		return -ENOENT;
-	if (folio_test_hwpoison(folio) || !folio_test_uptodate(folio)) {
+	if (kvm_gmem_is_pfn_mode(gi)) {
+		r = kvm_gmem_export_pfn(gi, index, pfn, max_order, attrs);
+		if (r)
+			return r;
+	} else {
+		/* A folio that was never zeroed is a hole. */
+		folio = filemap_get_folio(inode->i_mapping, index);
+		if (IS_ERR(folio))
+			return -ENOENT;
+		if (folio_test_hwpoison(folio) || !folio_test_uptodate(folio)) {
+			folio_put(folio);
+			return -ENOENT;
+		}
+		*pfn = folio_file_pfn(folio, index);
+		*max_order = 0;
+		*attrs = 0;
 		folio_put(folio);
-		return -ENOENT;
 	}
-	*pfn = folio_file_pfn(folio, index);
-	*max_order = 0;
-	*attrs = 0;
-	folio_put(folio);
 
 	/* Nor may the block hold a private page. */
 	while (*max_order &&
@@ -2042,6 +2117,8 @@ static struct inode *kvm_gmem_alloc_inode(struct super_block *sb)
 	gi->provider_ops = NULL;
 	INIT_LIST_HEAD(&gi->gmem_file_list);
 	INIT_LIST_HEAD(&gi->exports);
+	gi->punch_start = 0;
+	gi->punch_end = 0;
 	return &gi->vfs_inode;
 }
 
-- 
2.47.3

[RFC PATCH 06/14] KVM: guest_memfd: Report the size of holes in export lookups

[Fred Griffoul]

From: Fred Griffoul <fgriffo@amazon.co.uk>

A lookup reports a hole as a single page, so a consumer walks a large
private or unbacked range one page at a time, and cannot map a large
hole with one entry, e.g. to a scratch page.

Report the largest aligned hole around the page, up to the order the
caller asks for. The search checks one half-block at a time, so it
costs about one lookup per page of the hole it finds.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Fred Griffoul <fgriffo@amazon.co.uk>
---
 virt/kvm/guest_memfd.c | 48 ++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 46 insertions(+), 2 deletions(-)

diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 015f6657c386..1c5826c5384e 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -1417,6 +1417,48 @@ static int __kvm_gmem_export_lookup(struct inode *inode, pgoff_t index,
 	return 0;
 }
 
+/* Does no page of [start, start + nr) have a frame to map? */
+static bool kvm_gmem_export_is_hole(struct inode *inode, pgoff_t start,
+				    pgoff_t nr)
+{
+	unsigned long pfn;
+	pgoff_t index;
+	u32 attrs;
+	int order;
+
+	for (index = start; index < start + nr; index++) {
+		order = 0;
+		if (__kvm_gmem_export_lookup(inode, index, &pfn, &order,
+					     &attrs) != -ENOENT)
+			return false;
+		cond_resched();
+	}
+	return true;
+}
+
+/*
+ * The order, up to @max_order, of the largest naturally aligned block around
+ * @index that lies inside the file and has no frame to map.  Each step checks
+ * only the half of the larger block that does not hold @index.
+ */
+static int kvm_gmem_export_hole_order(struct inode *inode, pgoff_t index,
+				      int max_order)
+{
+	pgoff_t npages = i_size_read(inode) >> PAGE_SHIFT;
+	int order;
+
+	for (order = 0; order < max_order; order++) {
+		pgoff_t n = 1UL << order;
+		pgoff_t first = round_down(index, 2 * n);
+
+		if (first + 2 * n > npages ||
+		    !kvm_gmem_export_is_hole(inode, index & n ? first : first + n,
+					     n))
+			break;
+	}
+	return order;
+}
+
 /**
  * kvm_gmem_export_lookup_pfn - The frame behind a shared page.
  * @e:         The export.
@@ -1424,7 +1466,8 @@ static int __kvm_gmem_export_lookup(struct inode *inode, pgoff_t index,
  * @pfn:       The frame.
  * @max_order: In: the largest order the caller can map.  Out: the order of
  *             the naturally aligned block around @index that is contiguous,
- *             shared and of the same @attrs; 0 for -ENOENT.
+ *             shared and of the same @attrs or, for -ENOENT, that has no
+ *             frame to map at all.
  * @attrs:     The GUEST_MEMFD_PFN_* attributes of the block.
  *
  * Takes no reference and allocates nothing, and may be called from the
@@ -1438,6 +1481,7 @@ int kvm_gmem_export_lookup_pfn(struct guest_memfd_export *e, pgoff_t index,
 			       unsigned long *pfn, int *max_order, u32 *attrs)
 {
 	struct inode *inode = file_inode(e->file);
+	int order = *max_order;
 	int r;
 
 	if (index >= i_size_read(inode) >> PAGE_SHIFT)
@@ -1445,7 +1489,7 @@ int kvm_gmem_export_lookup_pfn(struct guest_memfd_export *e, pgoff_t index,
 
 	r = __kvm_gmem_export_lookup(inode, index, pfn, max_order, attrs);
 	if (r == -ENOENT)
-		*max_order = 0;
+		*max_order = kvm_gmem_export_hole_order(inode, index, order);
 	return r;
 }
 EXPORT_SYMBOL_GPL(kvm_gmem_export_lookup_pfn);
-- 
2.47.3

[RFC PATCH 07/14] KVM: guest_memfd: Update exports when pages are allocated

[Fred Griffoul]

From: Fred Griffoul <fgriffo@amazon.co.uk>

An export maps only the pages that have memory when it looks them up,
and a device cannot fault in the others, so guest_memfd must call
update() when a page gets memory.

Lookups treat a folio that was never zeroed as a hole. Zero folios in
one helper, under the invalidate lock, from the host fault, KVM's
get_pfn() and fallocate(), which did not zero at all, and update the
exports for each folio zeroed. Mark zeroed folios dirty: lookups hold no
reference, and memory-failure handling evicts clean folios that nothing
references.

In PFN mode, update the block that get_pfn() reports when it fills a
hole, since the provider may back all of it.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Fred Griffoul <fgriffo@amazon.co.uk>
---
 include/linux/guest_memfd.h | 26 ++++++------
 virt/kvm/guest_memfd.c      | 85 +++++++++++++++++++++++++++++++------
 2 files changed, 85 insertions(+), 26 deletions(-)

diff --git a/include/linux/guest_memfd.h b/include/linux/guest_memfd.h
index 937693ae9573..f8e89723dd68 100644
--- a/include/linux/guest_memfd.h
+++ b/include/linux/guest_memfd.h
@@ -90,13 +90,14 @@ static inline int kvm_gmem_revoke(struct inode *inode, pgoff_t start, size_t nr_
  * nor fault them.  It attaches to the file, looks up the frame behind each
  * shared page without taking a reference, and guest_memfd calls its update()
  * whenever what a lookup returns for a range may have changed: conversions,
- * punch holes, memory failure and a provider's kvm_gmem_revoke().  update()
- * makes the consumer's mappings of the range match what a lookup returns now:
- * a frame, or a hole for a private page or a page without a frame.  It may be
- * called for pages that did not change, and for ranges that do not follow the
- * blocks that lookups report: the consumer may then remap whole blocks, also
- * outside the range, e.g. to merge them into larger mappings.  A consumer that
- * can fault may instead only remove its mappings of the range.
+ * punch holes, memory failure, a provider's kvm_gmem_revoke(), fallocate() and
+ * pages getting their first frame.  update() makes the consumer's mappings of
+ * the range match what a lookup returns now: a frame, or a hole for a private
+ * page or a page without a frame.  It may be called for pages that did not
+ * change, and for ranges that do not follow the blocks that lookups report:
+ * the consumer may then remap whole blocks, also outside the range, e.g. to
+ * merge them into larger mappings.  A consumer that can fault may instead only
+ * remove its mappings of the range.
  *
  * Until update() returns, the frames it replaces stay valid, so a consumer
  * may switch each mapping to its new target in one step; once it returns,
@@ -105,11 +106,12 @@ static inline int kvm_gmem_revoke(struct inode *inode, pgoff_t start, size_t nr_
  * beforehand.
  *
  * guest_memfd calls update() with its invalidate lock held, exclusive when it
- * takes frames away or pages convert and shared on memory failure, possibly
- * with a folio of the file locked; so update() may only call
- * kvm_gmem_export_lookup_pfn().  Under the shared lock, updates may run at the
- * same time, also for overlapping ranges.  The consumer holds a lock across a
- * lookup and the mapping it installs, which update() also takes.  Lock order:
+ * takes frames away or pages convert, and shared when pages only gain frames
+ * and on memory failure, possibly with a folio of the file locked; so update()
+ * may only call kvm_gmem_export_lookup_pfn().  Under the shared lock, updates
+ * may run at the same time, also for overlapping ranges.  The consumer holds a
+ * lock across a lookup and the mapping it installs, which update() also takes.
+ * Lock order:
  *
  *   PFN mode provider lock (across kvm_gmem_revoke())
  *     guest_memfd invalidate lock
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 1c5826c5384e..42bb5f263e23 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -286,7 +286,8 @@ static bool kvm_gmem_range_has_attributes(struct inode *inode,
  * There is no backing storage for the memory, so the folio will remain
  * up-to-date until it's removed.
  *
- * Ignore accessed, referenced, and dirty flags.  The memory is
+ * Ignore accessed and referenced flags; the dirty flag only keeps a zeroed
+ * folio in the page cache, see kvm_gmem_folio_zero().  The memory is
  * unevictable and there is no storage to write back to.
  */
 static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index)
@@ -430,7 +431,8 @@ static void kvm_gmem_invalidate_end(struct inode *inode, pgoff_t start,
 
 /*
  * What a lookup returns for [start, end) may have changed: exclusive lock when
- * frames were taken away or pages converted, at least shared on memory failure.
+ * frames were taken away or pages converted, at least shared when pages only
+ * gained frames and on memory failure.
  */
 static void kvm_gmem_update_exports(struct inode *inode, pgoff_t start,
 				    pgoff_t end)
@@ -590,6 +592,64 @@ int kvm_gmem_revoke(struct inode *inode, pgoff_t start, size_t nr_pages,
 }
 EXPORT_SYMBOL_GPL(kvm_gmem_revoke);
 
+/*
+ * Zero a folio on its first use.  Lookups find it from then on, with no
+ * reference held, so mark it dirty: memory-failure handling drops clean
+ * folios that nothing references from the page cache (mapping_evict_folio()).
+ * Called with the folio locked.  Returns true if the folio was zeroed.
+ */
+static bool kvm_gmem_folio_zero(struct folio *folio)
+{
+	if (folio_test_uptodate(folio))
+		return false;
+
+	folio_zero_range(folio, 0, folio_size(folio));
+	folio_mark_uptodate(folio);
+	folio_mark_dirty(folio);
+	return true;
+}
+
+/* Zero a folio on its first use, and tell the exports.  Under the lock. */
+static void kvm_gmem_folio_clear(struct inode *inode, struct folio *folio)
+{
+	if (kvm_gmem_folio_zero(folio))
+		kvm_gmem_update_exports(inode, folio->index, folio_next_index(folio));
+}
+
+/*
+ * get_pfn() in PFN mode, which may back a hole: then tell the exports about
+ * the whole block that get_pfn() reports, as the provider may have backed all
+ * of it.  Called with the invalidate lock held.
+ */
+static int kvm_gmem_pfn_get(struct inode *inode, pgoff_t index,
+			    kvm_pfn_t *pfn, int *max_order, u32 *attrs)
+{
+	pgoff_t npages = i_size_read(inode) >> PAGE_SHIFT;
+	struct gmem_inode *gi = GMEM_I(inode);
+	bool hole = false;
+	unsigned long raw;
+	pgoff_t first;
+	int order, r;
+	size_t nr;
+
+	if (!list_empty(&gi->exports))
+		hole = gi->provider_ops->lookup_pfn_range(gi->provider, index,
+							  &raw, &nr) == -ENOENT;
+
+	r = gmem_provider_get_pfn(gi, index, pfn, &order, attrs);
+	if (r)
+		return r;
+
+	if (hole) {
+		first = round_down(index, 1UL << order);
+		kvm_gmem_update_exports(inode, first,
+					min(first + (1UL << order), npages));
+	}
+	if (max_order)
+		*max_order = order;
+	return 0;
+}
+
 static long kvm_gmem_allocate(struct inode *inode, loff_t offset, loff_t len)
 {
 	struct address_space *mapping = inode->i_mapping;
@@ -619,7 +679,7 @@ static long kvm_gmem_allocate(struct inode *inode, loff_t offset, loff_t len)
 				break;
 			}
 
-			r = gmem_provider_get_pfn(gi, index, &pfn, &max_order, NULL);
+			r = kvm_gmem_pfn_get(inode, index, &pfn, &max_order, NULL);
 			if (r)
 				break;
 
@@ -647,6 +707,8 @@ static long kvm_gmem_allocate(struct inode *inode, loff_t offset, loff_t len)
 
 		index = folio_next_index(folio);
 
+		/* Zeroed now, so that a device can map it before first use. */
+		kvm_gmem_folio_clear(inode, folio);
 		folio_unlock(folio);
 		folio_put(folio);
 
@@ -819,7 +881,7 @@ static vm_fault_t kvm_gmem_pfn_fault(struct vm_fault *vmf)
 	u32 attrs;
 	int r;
 
-	r = gmem_provider_get_pfn(GMEM_I(inode), vmf->pgoff, &pfn, NULL, &attrs);
+	r = kvm_gmem_pfn_get(inode, vmf->pgoff, &pfn, NULL, &attrs);
 	if (r)
 		return vmf_error(r);
 
@@ -873,6 +935,8 @@ static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf)
 		folio = kvm_gmem_get_folio(inode, vmf->pgoff);
 		if (IS_ERR(folio))
 			r = PTR_ERR(folio);
+		else if (!folio_test_large(folio))
+			kvm_gmem_folio_clear(inode, folio);
 	}
 	filemap_invalidate_unlock_shared(inode->i_mapping);
 
@@ -891,11 +955,6 @@ static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf)
 		goto out_folio;
 	}
 
-	if (!folio_test_uptodate(folio)) {
-		clear_highpage(folio_page(folio, 0));
-		folio_mark_uptodate(folio);
-	}
-
 	vmf->page = folio_file_page(folio, vmf->pgoff);
 
 out_folio:
@@ -1887,7 +1946,8 @@ static struct folio *__kvm_gmem_get_pfn(struct file *file,
 	}
 
 	if (kvm_gmem_is_pfn_mode(gi)) {
-		r = gmem_provider_get_pfn(gi, index, pfn, max_order, attrs);
+		r = kvm_gmem_pfn_get(file_inode(file), index, pfn, max_order,
+				     attrs);
 		if (r)
 			return ERR_PTR(r);
 		return NULL;
@@ -1903,10 +1963,7 @@ static struct folio *__kvm_gmem_get_pfn(struct file *file,
 		return ERR_PTR(-EHWPOISON);
 	}
 
-	if (!folio_test_uptodate(folio)) {
-		clear_highpage(folio_page(folio, 0));
-		folio_mark_uptodate(folio);
-	}
+	kvm_gmem_folio_clear(file_inode(file), folio);
 
 	*pfn = folio_file_pfn(folio, index);
 	if (max_order)
-- 
2.47.3

[RFC PATCH 08/14] iommufd: Track the domains of revocable pages in struct iopt_pages

[Fred Griffoul]

From: Fred Griffoul <fgriffo@amazon.co.uk>

guest_memfd pages, like dma-buf pages, can be taken away from iommufd
whatever references it holds, but the list of (area, domain) pairs to
unmap on a revoke lives in struct iopt_pages_dmabuf.

Move it to struct iopt_pages, rename the iopt_dmabuf_*track* helpers to
iopt_pages_*, and add iopt_pages_tracked() for the page types that need
it, so that other page types can use it. No functional change.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Fred Griffoul <fgriffo@amazon.co.uk>
---
 drivers/iommu/iommufd/io_pagetable.c | 24 +++++-----
 drivers/iommu/iommufd/io_pagetable.h | 33 +++++++++-----
 drivers/iommu/iommufd/pages.c        | 67 ++++++++++++++--------------
 3 files changed, 67 insertions(+), 57 deletions(-)

diff --git a/drivers/iommu/iommufd/io_pagetable.c b/drivers/iommu/iommufd/io_pagetable.c
index 4e447ce74cf6..43d518e4ce58 100644
--- a/drivers/iommu/iommufd/io_pagetable.c
+++ b/drivers/iommu/iommufd/io_pagetable.c
@@ -1086,14 +1086,14 @@ static void iopt_unfill_domain(struct io_pagetable *iopt,
 				WARN_ON(!area->storage_domain);
 			if (area->storage_domain == domain)
 				area->storage_domain = storage_domain;
-			if (iopt_is_dmabuf(pages)) {
+			if (iopt_pages_tracked(pages)) {
 				if (!iopt_dmabuf_revoked(pages))
 					iopt_area_unmap_domain(area, domain);
-				iopt_dmabuf_untrack_domain(pages, area, domain);
+				iopt_pages_untrack_domain(pages, area, domain);
 			}
 			mutex_unlock(&pages->mutex);
 
-			if (!iopt_is_dmabuf(pages))
+			if (!iopt_pages_tracked(pages))
 				iopt_area_unmap_domain(area, domain);
 		}
 		return;
@@ -1111,8 +1111,8 @@ static void iopt_unfill_domain(struct io_pagetable *iopt,
 		WARN_ON(area->storage_domain != domain);
 		area->storage_domain = NULL;
 		iopt_area_unfill_domain(area, pages, domain);
-		if (iopt_is_dmabuf(pages))
-			iopt_dmabuf_untrack_domain(pages, area, domain);
+		if (iopt_pages_tracked(pages))
+			iopt_pages_untrack_domain(pages, area, domain);
 		mutex_unlock(&pages->mutex);
 	}
 }
@@ -1143,15 +1143,15 @@ static int iopt_fill_domain(struct io_pagetable *iopt,
 			continue;
 
 		guard(mutex)(&pages->mutex);
-		if (iopt_is_dmabuf(pages)) {
-			rc = iopt_dmabuf_track_domain(pages, area, domain);
+		if (iopt_pages_tracked(pages)) {
+			rc = iopt_pages_track_domain(pages, area, domain);
 			if (rc)
 				goto out_unfill;
 		}
 		rc = iopt_area_fill_domain(area, domain);
 		if (rc) {
-			if (iopt_is_dmabuf(pages))
-				iopt_dmabuf_untrack_domain(pages, area, domain);
+			if (iopt_pages_tracked(pages))
+				iopt_pages_untrack_domain(pages, area, domain);
 			goto out_unfill;
 		}
 		if (!area->storage_domain) {
@@ -1180,8 +1180,8 @@ static int iopt_fill_domain(struct io_pagetable *iopt,
 			area->storage_domain = NULL;
 		}
 		iopt_area_unfill_domain(area, pages, domain);
-		if (iopt_is_dmabuf(pages))
-			iopt_dmabuf_untrack_domain(pages, area, domain);
+		if (iopt_pages_tracked(pages))
+			iopt_pages_untrack_domain(pages, area, domain);
 		mutex_unlock(&pages->mutex);
 	}
 	return rc;
@@ -1393,7 +1393,7 @@ static int iopt_area_split(struct iopt_area *area, unsigned long iova)
 		return -EBUSY;
 
 	/* Maintaining the domains_itree below is a bit complicated */
-	if (iopt_is_dmabuf(pages))
+	if (iopt_pages_tracked(pages))
 		return -EOPNOTSUPP;
 
 	if (new_start & (alignment - 1) ||
diff --git a/drivers/iommu/iommufd/io_pagetable.h b/drivers/iommu/iommufd/io_pagetable.h
index 27e3e311d395..0dddfb5db5fd 100644
--- a/drivers/iommu/iommufd/io_pagetable.h
+++ b/drivers/iommu/iommufd/io_pagetable.h
@@ -70,15 +70,14 @@ void iopt_area_unfill_domain(struct iopt_area *area, struct iopt_pages *pages,
 void iopt_area_unmap_domain(struct iopt_area *area,
 			    struct iommu_domain *domain);
 
-int iopt_dmabuf_track_domain(struct iopt_pages *pages, struct iopt_area *area,
-			     struct iommu_domain *domain);
-void iopt_dmabuf_untrack_domain(struct iopt_pages *pages,
-				struct iopt_area *area,
-				struct iommu_domain *domain);
-int iopt_dmabuf_track_all_domains(struct iopt_area *area,
-				  struct iopt_pages *pages);
-void iopt_dmabuf_untrack_all_domains(struct iopt_area *area,
-				     struct iopt_pages *pages);
+int iopt_pages_track_domain(struct iopt_pages *pages, struct iopt_area *area,
+			    struct iommu_domain *domain);
+void iopt_pages_untrack_domain(struct iopt_pages *pages, struct iopt_area *area,
+			       struct iommu_domain *domain);
+int iopt_pages_track_all_domains(struct iopt_area *area,
+				 struct iopt_pages *pages);
+void iopt_pages_untrack_all_domains(struct iopt_area *area,
+				    struct iopt_pages *pages);
 
 static inline unsigned long iopt_area_index(struct iopt_area *area)
 {
@@ -194,7 +193,8 @@ enum iopt_address_type {
 	IOPT_ADDRESS_DMABUF,
 };
 
-struct iopt_pages_dmabuf_track {
+/* One (area, domain) mapping of revocable pages, on iopt_pages::tracker. */
+struct iopt_pages_track {
 	struct iommu_domain *domain;
 	struct iopt_area *area;
 	struct list_head elm;
@@ -205,7 +205,6 @@ struct iopt_pages_dmabuf {
 	struct phys_vec phys;
 	/* Always PAGE_SIZE aligned */
 	unsigned long start;
-	struct list_head tracker;
 };
 
 /*
@@ -246,6 +245,12 @@ struct iopt_pages {
 	struct rb_root_cached access_itree;
 	/* Of iopt_area::pages_node */
 	struct rb_root_cached domains_itree;
+	/*
+	 * Of iopt_pages_track::elm.  The owner of these pages can revoke them
+	 * whatever references iommufd holds, so every (area, domain) that
+	 * maps them is listed here for the revoke.  See iopt_pages_tracked().
+	 */
+	struct list_head tracker;
 };
 
 static inline bool iopt_is_dmabuf(struct iopt_pages *pages)
@@ -255,6 +260,12 @@ static inline bool iopt_is_dmabuf(struct iopt_pages *pages)
 	return pages->type == IOPT_ADDRESS_DMABUF;
 }
 
+/* The owner of the pages can revoke them, so their domains are tracked. */
+static inline bool iopt_pages_tracked(struct iopt_pages *pages)
+{
+	return iopt_is_dmabuf(pages);
+}
+
 static inline bool iopt_dmabuf_revoked(struct iopt_pages *pages)
 {
 	lockdep_assert_held(&pages->mutex);
diff --git a/drivers/iommu/iommufd/pages.c b/drivers/iommu/iommufd/pages.c
index 404f31d8f729..cf0f08c4e9a6 100644
--- a/drivers/iommu/iommufd/pages.c
+++ b/drivers/iommu/iommufd/pages.c
@@ -1384,6 +1384,7 @@ static struct iopt_pages *iopt_alloc_pages(unsigned long start_byte,
 	pages->npages = DIV_ROUND_UP(length + start_byte, PAGE_SIZE);
 	pages->access_itree = RB_ROOT_CACHED;
 	pages->domains_itree = RB_ROOT_CACHED;
+	INIT_LIST_HEAD(&pages->tracker);
 	pages->writable = writable;
 	if (capable(CAP_IPC_LOCK))
 		pages->account_mode = IOPT_PAGES_ACCOUNT_NONE;
@@ -1434,13 +1435,13 @@ struct iopt_pages *iopt_alloc_file_pages(struct file *file,
 static void iopt_revoke_notify(struct dma_buf_attachment *attach)
 {
 	struct iopt_pages *pages = attach->importer_priv;
-	struct iopt_pages_dmabuf_track *track;
+	struct iopt_pages_track *track;
 
 	guard(mutex)(&pages->mutex);
 	if (iopt_dmabuf_revoked(pages))
 		return;
 
-	list_for_each_entry(track, &pages->dmabuf.tracker, elm) {
+	list_for_each_entry(track, &pages->tracker, elm) {
 		struct iopt_area *area = track->area;
 
 		iopt_area_unmap_domain_range(area, track->domain,
@@ -1557,7 +1558,6 @@ struct iopt_pages *iopt_alloc_dmabuf_pages(struct iommufd_ctx *ictx,
 	pages->account_mode = IOPT_PAGES_ACCOUNT_NONE;
 	pages->type = IOPT_ADDRESS_DMABUF;
 	pages->dmabuf.start = start - start_byte;
-	INIT_LIST_HEAD(&pages->dmabuf.tracker);
 
 	rc = iopt_map_dmabuf(ictx, pages, dmabuf);
 	if (rc) {
@@ -1568,16 +1568,16 @@ struct iopt_pages *iopt_alloc_dmabuf_pages(struct iommufd_ctx *ictx,
 	return pages;
 }
 
-int iopt_dmabuf_track_domain(struct iopt_pages *pages, struct iopt_area *area,
-			     struct iommu_domain *domain)
+int iopt_pages_track_domain(struct iopt_pages *pages, struct iopt_area *area,
+			    struct iommu_domain *domain)
 {
-	struct iopt_pages_dmabuf_track *track;
+	struct iopt_pages_track *track;
 
 	lockdep_assert_held(&pages->mutex);
-	if (WARN_ON(!iopt_is_dmabuf(pages)))
+	if (WARN_ON(!iopt_pages_tracked(pages)))
 		return -EINVAL;
 
-	list_for_each_entry(track, &pages->dmabuf.tracker, elm)
+	list_for_each_entry(track, &pages->tracker, elm)
 		if (WARN_ON(track->domain == domain && track->area == area))
 			return -EINVAL;
 
@@ -1586,21 +1586,20 @@ int iopt_dmabuf_track_domain(struct iopt_pages *pages, struct iopt_area *area,
 		return -ENOMEM;
 	track->domain = domain;
 	track->area = area;
-	list_add_tail(&track->elm, &pages->dmabuf.tracker);
+	list_add_tail(&track->elm, &pages->tracker);
 
 	return 0;
 }
 
-void iopt_dmabuf_untrack_domain(struct iopt_pages *pages,
-				struct iopt_area *area,
-				struct iommu_domain *domain)
+void iopt_pages_untrack_domain(struct iopt_pages *pages, struct iopt_area *area,
+			       struct iommu_domain *domain)
 {
-	struct iopt_pages_dmabuf_track *track;
+	struct iopt_pages_track *track;
 
 	lockdep_assert_held(&pages->mutex);
-	WARN_ON(!iopt_is_dmabuf(pages));
+	WARN_ON(!iopt_pages_tracked(pages));
 
-	list_for_each_entry(track, &pages->dmabuf.tracker, elm) {
+	list_for_each_entry(track, &pages->tracker, elm) {
 		if (track->domain == domain && track->area == area) {
 			list_del(&track->elm);
 			kfree(track);
@@ -1610,36 +1609,36 @@ void iopt_dmabuf_untrack_domain(struct iopt_pages *pages,
 	WARN_ON(true);
 }
 
-int iopt_dmabuf_track_all_domains(struct iopt_area *area,
-				  struct iopt_pages *pages)
+int iopt_pages_track_all_domains(struct iopt_area *area,
+				 struct iopt_pages *pages)
 {
-	struct iopt_pages_dmabuf_track *track;
+	struct iopt_pages_track *track;
 	struct iommu_domain *domain;
 	unsigned long index;
 	int rc;
 
-	list_for_each_entry(track, &pages->dmabuf.tracker, elm)
+	list_for_each_entry(track, &pages->tracker, elm)
 		if (WARN_ON(track->area == area))
 			return -EINVAL;
 
 	xa_for_each(&area->iopt->domains, index, domain) {
-		rc = iopt_dmabuf_track_domain(pages, area, domain);
+		rc = iopt_pages_track_domain(pages, area, domain);
 		if (rc)
 			goto err_untrack;
 	}
 	return 0;
 err_untrack:
-	iopt_dmabuf_untrack_all_domains(area, pages);
+	iopt_pages_untrack_all_domains(area, pages);
 	return rc;
 }
 
-void iopt_dmabuf_untrack_all_domains(struct iopt_area *area,
-				     struct iopt_pages *pages)
+void iopt_pages_untrack_all_domains(struct iopt_area *area,
+				    struct iopt_pages *pages)
 {
-	struct iopt_pages_dmabuf_track *track;
-	struct iopt_pages_dmabuf_track *tmp;
+	struct iopt_pages_track *track;
+	struct iopt_pages_track *tmp;
 
-	list_for_each_entry_safe(track, tmp, &pages->dmabuf.tracker,
+	list_for_each_entry_safe(track, tmp, &pages->tracker,
 				 elm) {
 		if (track->area == area) {
 			list_del(&track->elm);
@@ -1656,6 +1655,7 @@ void iopt_release_pages(struct kref *kref)
 	WARN_ON(!RB_EMPTY_ROOT(&pages->domains_itree.rb_root));
 	WARN_ON(pages->npinned);
 	WARN_ON(!xa_empty(&pages->pinned_pfns));
+	WARN_ON(!list_empty(&pages->tracker));
 	if (iopt_is_dmabuf(pages) && pages->dmabuf.attach) {
 		struct dma_buf *dmabuf = pages->dmabuf.attach->dmabuf;
 
@@ -1664,7 +1664,6 @@ void iopt_release_pages(struct kref *kref)
 		dma_resv_unlock(dmabuf->resv);
 		dma_buf_detach(dmabuf, pages->dmabuf.attach);
 		dma_buf_put(dmabuf);
-		WARN_ON(!list_empty(&pages->dmabuf.tracker));
 	} else if (pages->type == IOPT_ADDRESS_FILE) {
 		fput(pages->file);
 	}
@@ -1749,7 +1748,7 @@ static void __iopt_area_unfill_domain(struct iopt_area *area,
 
 	lockdep_assert_held(&pages->mutex);
 
-	if (iopt_is_dmabuf(pages)) {
+	if (iopt_pages_tracked(pages)) {
 		if (WARN_ON(iopt_dmabuf_revoked(pages)))
 			return;
 		iopt_area_unmap_domain_range(area, domain, start_index,
@@ -1917,8 +1916,8 @@ int iopt_area_fill_domains(struct iopt_area *area, struct iopt_pages *pages)
 		return 0;
 
 	mutex_lock(&pages->mutex);
-	if (iopt_is_dmabuf(pages)) {
-		rc = iopt_dmabuf_track_all_domains(area, pages);
+	if (iopt_pages_tracked(pages)) {
+		rc = iopt_pages_track_all_domains(area, pages);
 		if (rc)
 			goto out_unlock;
 	}
@@ -1983,8 +1982,8 @@ int iopt_area_fill_domains(struct iopt_area *area, struct iopt_pages *pages)
 	}
 	pfn_reader_destroy(&pfns);
 out_untrack:
-	if (iopt_is_dmabuf(pages))
-		iopt_dmabuf_untrack_all_domains(area, pages);
+	if (iopt_pages_tracked(pages))
+		iopt_pages_untrack_all_domains(area, pages);
 out_unlock:
 	mutex_unlock(&pages->mutex);
 	return rc;
@@ -2024,8 +2023,8 @@ void iopt_area_unfill_domains(struct iopt_area *area, struct iopt_pages *pages)
 		WARN_ON(RB_EMPTY_NODE(&area->pages_node.rb));
 	interval_tree_remove(&area->pages_node, &pages->domains_itree);
 	iopt_area_unfill_domain(area, pages, area->storage_domain);
-	if (iopt_is_dmabuf(pages))
-		iopt_dmabuf_untrack_all_domains(area, pages);
+	if (iopt_pages_tracked(pages))
+		iopt_pages_untrack_all_domains(area, pages);
 	area->storage_domain = NULL;
 out_unlock:
 	mutex_unlock(&pages->mutex);
-- 
2.47.3

[RFC PATCH 09/14] iommufd: Map guest_memfd files

[Fred Griffoul]

From: Fred Griffoul <fgriffo@amazon.co.uk>

A VMM that assigns a device to a guest backed by guest_memfd needs the
IOMMU mappings to follow the guest's shared memory. IOMMU_IOAS_MAP_FILE
pins what it maps, which guest_memfd does not allow.

Accept a guest_memfd in IOMMU_IOAS_MAP_FILE through guest_memfd's export
interface: its pages are not pinned, and each update() unmaps the range
and maps what a lookup returns now, read-only frames without write
permission. This is a demonstration: entries are PAGE_SIZE, a device
that touches the range during an update faults, and a range that fails
to map again stays unmapped, with a warning.

Dirty tracking, the no-IOMMU mode, kernel access, splitting an area and
a frame at address 0 are refused, since iommufd could not follow them.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Fred Griffoul <fgriffo@amazon.co.uk>
---
 drivers/iommu/iommufd/io_pagetable.c |  14 +-
 drivers/iommu/iommufd/io_pagetable.h |  30 ++-
 drivers/iommu/iommufd/pages.c        | 299 ++++++++++++++++++++++++++-
 include/uapi/linux/iommufd.h         |  11 +-
 4 files changed, 349 insertions(+), 5 deletions(-)

diff --git a/drivers/iommu/iommufd/io_pagetable.c b/drivers/iommu/iommufd/io_pagetable.c
index 43d518e4ce58..62fa553cdffc 100644
--- a/drivers/iommu/iommufd/io_pagetable.c
+++ b/drivers/iommu/iommufd/io_pagetable.c
@@ -217,9 +217,13 @@ static int iopt_insert_area(struct io_pagetable *iopt, struct iopt_area *area,
 		return -EPERM;
 
 	area->iommu_prot = iommu_prot;
+	area->gmem = iopt_is_gmem(pages);
 	area->page_offset = start_byte % PAGE_SIZE;
 	if (area->page_offset & (iopt->iova_alignment - 1))
 		return -EINVAL;
+	/* guest_memfd frames are mapped whole, see iopt_gmem_map(). */
+	if (area->gmem && (area->page_offset || !PAGE_ALIGNED(length)))
+		return -EINVAL;
 
 	area->node.start = iova;
 	if (check_add_overflow(iova, length - 1, &area->node.last))
@@ -289,6 +293,9 @@ static int iopt_alloc_area_pages(struct io_pagetable *iopt,
 		case IOPT_ADDRESS_DMABUF:
 			start = elm->start_byte + elm->pages->dmabuf.start;
 			break;
+		case IOPT_ADDRESS_GMEM:
+			start = elm->start_byte + elm->pages->gmem.start;
+			break;
 		}
 		rc = iopt_alloc_iova(iopt, dst_iova, start, length);
 		if (rc)
@@ -515,8 +522,13 @@ int iopt_map_file_pages(struct iommufd_ctx *ictx, struct io_pagetable *iopt,
 		if (!file)
 			return -EBADF;
 
-		pages = iopt_alloc_file_pages(file, start_byte, start, length,
+		/* -ENODEV: not a guest_memfd, so map it as a memfd. */
+		pages = iopt_alloc_gmem_pages(file, start, length,
 					      iommu_prot & IOMMU_WRITE);
+		if (PTR_ERR(pages) == -ENODEV)
+			pages = iopt_alloc_file_pages(file, start_byte, start,
+						      length,
+						      iommu_prot & IOMMU_WRITE);
 		fput(file);
 		if (IS_ERR(pages))
 			return PTR_ERR(pages);
diff --git a/drivers/iommu/iommufd/io_pagetable.h b/drivers/iommu/iommufd/io_pagetable.h
index 0dddfb5db5fd..11e16d22436a 100644
--- a/drivers/iommu/iommufd/io_pagetable.h
+++ b/drivers/iommu/iommufd/io_pagetable.h
@@ -6,6 +6,7 @@
 #define __IO_PAGETABLE_H
 
 #include <linux/dma-buf.h>
+#include <linux/guest_memfd.h>
 #include <linux/interval_tree.h>
 #include <linux/kref.h>
 #include <linux/mutex.h>
@@ -13,6 +14,8 @@
 
 #include "iommufd_private.h"
 
+struct mem_cgroup;
+
 struct iommu_domain;
 
 /*
@@ -48,6 +51,8 @@ struct iopt_area {
 	/* IOMMU_READ, IOMMU_WRITE, etc */
 	int iommu_prot;
 	bool prevent_access : 1;
+	/* The pages come from a guest_memfd and may have holes */
+	bool gmem : 1;
 	unsigned int num_accesses;
 	unsigned int num_locks;
 };
@@ -191,6 +196,7 @@ enum iopt_address_type {
 	IOPT_ADDRESS_USER = 0,
 	IOPT_ADDRESS_FILE,
 	IOPT_ADDRESS_DMABUF,
+	IOPT_ADDRESS_GMEM,
 };
 
 /* One (area, domain) mapping of revocable pages, on iopt_pages::tracker. */
@@ -207,6 +213,17 @@ struct iopt_pages_dmabuf {
 	unsigned long start;
 };
 
+struct iopt_pages_gmem {
+	struct guest_memfd_export export;
+	/* From symbol_get(): KVM may be a module */
+	typeof(kvm_gmem_export_lookup_pfn) *lookup_pfn;
+	typeof(kvm_gmem_export_detach) *detach;
+	/* Byte offset in the guest_memfd, always PAGE_SIZE aligned */
+	unsigned long start;
+	/* Charged for the page tables that a fill allocates */
+	struct mem_cgroup *memcg;
+};
+
 /*
  * This holds a pinned page list for multiple areas of IO address space. The
  * pages always originate from a linear chunk of userspace VA. Multiple
@@ -236,6 +253,8 @@ struct iopt_pages {
 		};
 		/* IOPT_ADDRESS_DMABUF */
 		struct iopt_pages_dmabuf dmabuf;
+		/* IOPT_ADDRESS_GMEM */
+		struct iopt_pages_gmem gmem;
 	};
 	bool writable:1;
 	u8 account_mode;
@@ -260,10 +279,15 @@ static inline bool iopt_is_dmabuf(struct iopt_pages *pages)
 	return pages->type == IOPT_ADDRESS_DMABUF;
 }
 
+static inline bool iopt_is_gmem(struct iopt_pages *pages)
+{
+	return pages->type == IOPT_ADDRESS_GMEM;
+}
+
 /* The owner of the pages can revoke them, so their domains are tracked. */
 static inline bool iopt_pages_tracked(struct iopt_pages *pages)
 {
-	return iopt_is_dmabuf(pages);
+	return iopt_is_dmabuf(pages) || iopt_is_gmem(pages);
 }
 
 static inline bool iopt_dmabuf_revoked(struct iopt_pages *pages)
@@ -285,6 +309,10 @@ struct iopt_pages *iopt_alloc_dmabuf_pages(struct iommufd_ctx *ictx,
 					   unsigned long start_byte,
 					   unsigned long start,
 					   unsigned long length, bool writable);
+struct iopt_pages *iopt_alloc_gmem_pages(struct file *file,
+					 unsigned long start,
+					 unsigned long length,
+					 bool writable);
 void iopt_release_pages(struct kref *kref);
 static inline void iopt_put_pages(struct iopt_pages *pages)
 {
diff --git a/drivers/iommu/iommufd/pages.c b/drivers/iommu/iommufd/pages.c
index cf0f08c4e9a6..88354aa554cf 100644
--- a/drivers/iommu/iommufd/pages.c
+++ b/drivers/iommu/iommufd/pages.c
@@ -52,6 +52,7 @@
 #include <linux/iommu.h>
 #include <linux/iommufd.h>
 #include <linux/kthread.h>
+#include <linux/memcontrol.h>
 #include <linux/overflow.h>
 #include <linux/slab.h>
 #include <linux/sched/mm.h>
@@ -238,6 +239,40 @@ static void iommu_unmap_nofail(struct iommu_domain *domain, unsigned long iova,
 	WARN_ON(ret != size);
 }
 
+/*
+ * guest_memfd pages are mapped with PAGE_SIZE entries, and may have holes.
+ * iommu_unmap() stops at the first IOVA that is not mapped, which
+ * iommu_unmap_nofail() warns about, so find the runs that are mapped and
+ * unmap each one, without touching the holes.
+ * iopt_gmem_map() refuses frame 0, which iova_to_phys() reports for
+ * an IOVA that is not mapped.
+ */
+static void iopt_gmem_unmap(struct iopt_area *area,
+			    struct iommu_domain *domain,
+			    unsigned long start_index,
+			    unsigned long last_index)
+{
+	unsigned long iova = iopt_area_index_to_iova(area, start_index);
+	size_t left = (last_index - start_index + 1) * PAGE_SIZE;
+
+	/* A guest_memfd can be large: yield, once per page scanned. */
+	while (left) {
+		size_t len = 0;
+
+		while (len < left && iommu_iova_to_phys(domain, iova + len)) {
+			len += PAGE_SIZE;
+			cond_resched();
+		}
+		if (len)
+			iommu_unmap_nofail(domain, iova, len);
+		/* Step over the run and the hole page that ended it. */
+		len = min(len + PAGE_SIZE, left);
+		iova += len;
+		left -= len;
+		cond_resched();
+	}
+}
+
 static void iopt_area_unmap_domain_range(struct iopt_area *area,
 					 struct iommu_domain *domain,
 					 unsigned long start_index,
@@ -245,6 +280,11 @@ static void iopt_area_unmap_domain_range(struct iopt_area *area,
 {
 	unsigned long start_iova = iopt_area_index_to_iova(area, start_index);
 
+	if (area->gmem) {
+		iopt_gmem_unmap(area, domain, start_index, last_index);
+		return;
+	}
+
 	iommu_unmap_nofail(domain, start_iova,
 			   iopt_area_index_to_iova_last(area, last_index) -
 				   start_iova + 1);
@@ -1349,6 +1389,10 @@ static int pfn_reader_first(struct pfn_reader *pfns, struct iopt_pages *pages,
 	    WARN_ON(last_index < start_index))
 		return -EINVAL;
 
+	/* guest_memfd pages are read from guest_memfd, see iopt_gmem_map() */
+	if (WARN_ON(iopt_is_gmem(pages)))
+		return -EINVAL;
+
 	rc = pfn_reader_init(pfns, pages, start_index, last_index);
 	if (rc)
 		return rc;
@@ -1647,6 +1691,230 @@ void iopt_pages_untrack_all_domains(struct iopt_area *area,
 	}
 }
 
+static int iopt_gmem_prot(struct iopt_area *area, u32 attrs)
+{
+	int prot = area->iommu_prot;
+
+	if (attrs & GUEST_MEMFD_PFN_READONLY)
+		prot &= ~IOMMU_WRITE;
+	return prot;
+}
+
+/*
+ * Map what guest_memfd backs in [start_index, last_index] of the area into
+ * @domain, with PAGE_SIZE entries, so that updating part of a large block
+ * never has to cut a larger IOMMU page.  On failure, nothing in the range is
+ * mapped.
+ */
+static int iopt_gmem_map(struct iopt_area *area, struct iopt_pages *pages,
+			 struct iommu_domain *domain, unsigned long start_index,
+			 unsigned long last_index)
+{
+	struct guest_memfd_export *e = &pages->gmem.export;
+	unsigned long base = pages->gmem.start >> PAGE_SHIFT;
+	unsigned long index = start_index;
+	int rc = 0;
+
+	lockdep_assert_held(&pages->mutex);
+
+	/* A revoke would race a dirty bitmap read and lose its dirty bits. */
+	if (domain->dirty_ops)
+		return -EOPNOTSUPP;
+
+	/* The device of a no-IOMMU domain keeps the addresses after a revoke. */
+	if (IS_ENABLED(CONFIG_IOMMUFD_NOIOMMU) &&
+	    domain->owner == &iommufd_noiommu_ops)
+		return -EOPNOTSUPP;
+
+	while (index <= last_index) {
+		unsigned long pfn, iova, end, nr;
+		/* No larger than the rest of the range: holes cost per page. */
+		int order = min_t(int, PUD_ORDER, fls_long(last_index - index));
+		u32 attrs;
+		int prot;
+
+		cond_resched();
+
+		rc = pages->gmem.lookup_pfn(e, base + index, &pfn, &order,
+					    &attrs);
+		/* The aligned block of frames, or of holes, around index. */
+		end = ALIGN(base + index + 1, 1UL << order) - base;
+		nr = min(end, last_index + 1) - index;
+		if (rc == -ENOENT) {
+			index += nr;
+			continue;
+		}
+		/* Frame 0 would look unmapped to iopt_gmem_unmap(). */
+		if (!rc && !pfn)
+			rc = -EINVAL;
+		if (rc)
+			goto err_unmap;
+
+		iova = iopt_area_index_to_iova(area, index);
+		prot = iopt_gmem_prot(area, attrs);
+
+		/* A read-only frame in an area that is not readable: a hole. */
+		if (!(prot & (IOMMU_READ | IOMMU_WRITE))) {
+			index += nr;
+			continue;
+		}
+
+		rc = batch_iommu_map_small(domain, iova, PFN_PHYS(pfn),
+					   nr * PAGE_SIZE, prot);
+		if (rc)
+			goto err_unmap;
+		index += nr;
+	}
+	return 0;
+
+err_unmap:
+	if (index > start_index)
+		iopt_gmem_unmap(area, domain, start_index, index - 1);
+	return rc;
+}
+
+/* Map the whole area into every domain of its io_pagetable. */
+static int iopt_gmem_fill_domains(struct iopt_area *area,
+				  struct iopt_pages *pages)
+{
+	struct iommu_domain *domain, *undo;
+	unsigned long index, undo_index;
+	int rc;
+
+	xa_for_each(&area->iopt->domains, index, domain) {
+		rc = iopt_gmem_map(area, pages, domain, iopt_area_index(area),
+				   iopt_area_last_index(area));
+		if (rc)
+			goto err_unmap;
+	}
+	return 0;
+
+err_unmap:
+	xa_for_each(&area->iopt->domains, undo_index, undo) {
+		if (undo_index >= index)
+			break;
+		iopt_gmem_unmap(area, undo, iopt_area_index(area),
+				iopt_area_last_index(area));
+	}
+	return rc;
+}
+
+/*
+ * guest_memfd says that what [start, start + nr_pages) of the file holds may
+ * have changed: in every (area, domain) over part of it, unmap that part and
+ * map what guest_memfd backs now.  A device that accesses it in between, or
+ * after a failure to map, faults.
+ */
+static void iopt_gmem_update(struct guest_memfd_export *e, pgoff_t start,
+			     pgoff_t nr_pages)
+{
+	struct iopt_pages *pages =
+		container_of(e, struct iopt_pages, gmem.export);
+	pgoff_t base = pages->gmem.start >> PAGE_SHIFT;
+	struct iopt_pages_track *track;
+	struct mem_cgroup *old_memcg;
+	unsigned long first, last;
+
+	if (start + nr_pages <= base || start >= base + pages->npages)
+		return;
+	first = max(start, base) - base;
+	last = min(start + nr_pages, base + pages->npages) - base - 1;
+
+	guard(mutex)(&pages->mutex);
+	/* The page tables belong to whoever mapped the file, not to this task. */
+	old_memcg = set_active_memcg(pages->gmem.memcg);
+	list_for_each_entry(track, &pages->tracker, elm) {
+		struct iopt_area *area = track->area;
+		unsigned long s = max(first, iopt_area_index(area));
+		unsigned long l = min(last, iopt_area_last_index(area));
+
+		if (s > l)
+			continue;
+		iopt_gmem_unmap(area, track->domain, s, l);
+		if (iopt_gmem_map(area, pages, track->domain, s, l))
+			pr_warn_ratelimited("iommufd: guest_memfd pages left unmapped after an update\n");
+	}
+	set_active_memcg(old_memcg);
+}
+
+static const struct guest_memfd_export_ops iopt_gmem_export_ops = {
+	.update	= iopt_gmem_update,
+};
+
+/**
+ * iopt_alloc_gmem_pages() - Pages backed by a guest_memfd file
+ * @file: The file
+ * @start: Byte offset in the file
+ * @length: Number of bytes
+ * @writable: The pages may be mapped writable
+ *
+ * The pages are not pinned, and every domain that maps them follows
+ * guest_memfd's changes.
+ *
+ * Return: the pages, ERR_PTR(-ENODEV) if @file is not a guest_memfd file, or
+ * another ERR_PTR().
+ */
+struct iopt_pages *iopt_alloc_gmem_pages(struct file *file,
+					 unsigned long start,
+					 unsigned long length,
+					 bool writable)
+{
+	static struct lock_class_key pages_gmem_mutex_key;
+	int (*attach)(struct file *file, struct guest_memfd_export *e);
+	struct iopt_pages *pages;
+	unsigned long end;
+	int rc;
+
+	/* KVM may be a module, or not loaded: then this is no guest_memfd. */
+	attach = symbol_get(kvm_gmem_export_attach);
+	if (!attach)
+		return ERR_PTR(-ENODEV);
+
+	pages = iopt_alloc_pages(0, length, writable);
+	if (IS_ERR(pages))
+		goto out_put;
+
+	/*
+	 * Pinned pages take the mmap_lock under their mutex.  The mutex of
+	 * guest_memfd pages is taken under guest_memfd's invalidate lock, which
+	 * nests in the mmap_lock of a host fault, so give it its own class.
+	 */
+	lockdep_set_class(&pages->mutex, &pages_gmem_mutex_key);
+
+	/* guest_memfd pages are not pinned, so they are not accounted. */
+	pages->account_mode = IOPT_PAGES_ACCOUNT_NONE;
+	pages->type = IOPT_ADDRESS_GMEM;
+	pages->gmem.start = start;
+	pages->gmem.export.ops = &iopt_gmem_export_ops;
+
+	rc = attach(file, &pages->gmem.export);
+	if (rc) {
+		iopt_put_pages(pages);
+		pages = ERR_PTR(rc);
+		goto out_put;
+	}
+
+	if (check_add_overflow(start, length, &end) ||
+	    end > i_size_read(file_inode(file))) {
+		rc = -EINVAL;
+		goto err_detach;
+	}
+
+	/* The file holds the KVM module from here on. */
+	pages->gmem.lookup_pfn = symbol_get(kvm_gmem_export_lookup_pfn);
+	pages->gmem.detach = symbol_get(kvm_gmem_export_detach);
+	pages->gmem.memcg = get_mem_cgroup_from_mm(current->mm);
+	goto out_put;
+
+err_detach:
+	pages->gmem.detach = symbol_get(kvm_gmem_export_detach);
+	iopt_put_pages(pages);
+	pages = ERR_PTR(rc);
+out_put:
+	symbol_put(kvm_gmem_export_attach);
+	return pages;
+}
+
 void iopt_release_pages(struct kref *kref)
 {
 	struct iopt_pages *pages = container_of(kref, struct iopt_pages, kref);
@@ -1664,6 +1932,14 @@ void iopt_release_pages(struct kref *kref)
 		dma_resv_unlock(dmabuf->resv);
 		dma_buf_detach(dmabuf, pages->dmabuf.attach);
 		dma_buf_put(dmabuf);
+	} else if (iopt_is_gmem(pages)) {
+		if (pages->gmem.detach) {
+			pages->gmem.detach(&pages->gmem.export);
+			symbol_put(kvm_gmem_export_detach);
+		}
+		if (pages->gmem.lookup_pfn)
+			symbol_put(kvm_gmem_export_lookup_pfn);
+		mem_cgroup_put(pages->gmem.memcg);
 	} else if (pages->type == IOPT_ADDRESS_FILE) {
 		fput(pages->file);
 	}
@@ -1814,6 +2090,12 @@ static void iopt_area_unfill_partial_domain(struct iopt_area *area,
  */
 void iopt_area_unmap_domain(struct iopt_area *area, struct iommu_domain *domain)
 {
+	if (area->gmem) {
+		iopt_gmem_unmap(area, domain, iopt_area_index(area),
+				iopt_area_last_index(area));
+		return;
+	}
+
 	iommu_unmap_nofail(domain, iopt_area_iova(area),
 			   iopt_area_length(area));
 }
@@ -1857,6 +2139,11 @@ int iopt_area_fill_domain(struct iopt_area *area, struct iommu_domain *domain)
 	if (iopt_dmabuf_revoked(area->pages))
 		return 0;
 
+	if (iopt_is_gmem(area->pages))
+		return iopt_gmem_map(area, area->pages, domain,
+				     iopt_area_index(area),
+				     iopt_area_last_index(area));
+
 	rc = pfn_reader_first(&pfns, area->pages, iopt_area_index(area),
 			      iopt_area_last_index(area));
 	if (rc)
@@ -1922,7 +2209,11 @@ int iopt_area_fill_domains(struct iopt_area *area, struct iopt_pages *pages)
 			goto out_unlock;
 	}
 
-	if (!iopt_dmabuf_revoked(pages)) {
+	if (iopt_is_gmem(pages)) {
+		rc = iopt_gmem_fill_domains(area, pages);
+		if (rc)
+			goto out_untrack;
+	} else if (!iopt_dmabuf_revoked(pages)) {
 		rc = pfn_reader_first(&pfns, pages, iopt_area_index(area),
 				      iopt_area_last_index(area));
 		if (rc)
@@ -2361,7 +2652,7 @@ int iopt_pages_rw_access(struct iopt_pages *pages, unsigned long start_byte,
 	if ((flags & IOMMUFD_ACCESS_RW_WRITE) && !pages->writable)
 		return -EPERM;
 
-	if (iopt_is_dmabuf(pages))
+	if (iopt_is_dmabuf(pages) || iopt_is_gmem(pages))
 		return -EINVAL;
 
 	if (pages->type != IOPT_ADDRESS_USER)
@@ -2453,6 +2744,10 @@ int iopt_area_add_access(struct iopt_area *area, unsigned long start_index,
 	if (iopt_is_dmabuf(pages))
 		return -EINVAL;
 
+	/* guest_memfd frames may have no struct page, and are not pinned. */
+	if (iopt_is_gmem(pages))
+		return -EINVAL;
+
 	mutex_lock(&pages->mutex);
 	access = iopt_pages_get_exact_access(pages, start_index, last_index);
 	if (access) {
diff --git a/include/uapi/linux/iommufd.h b/include/uapi/linux/iommufd.h
index 206fa667c782..db5f09c21d1f 100644
--- a/include/uapi/linux/iommufd.h
+++ b/include/uapi/linux/iommufd.h
@@ -252,7 +252,7 @@ struct iommu_ioas_noiommu_get_pa {
  * @size: sizeof(struct iommu_ioas_map_file)
  * @flags: same as for iommu_ioas_map
  * @ioas_id: same as for iommu_ioas_map
- * @fd: the memfd or supported dma-buf file to map
+ * @fd: the memfd, guest_memfd or supported dma-buf file to map
  * @start: byte offset from start of the file to map from
  * @length: same as for iommu_ioas_map
  * @iova: same as for iommu_ioas_map
@@ -263,6 +263,15 @@ struct iommu_ioas_noiommu_get_pa {
  * VFIO PCI dma-bufs exported through VFIO_DEVICE_FEATURE_DMA_BUF, and
  * other dma-bufs may be rejected. All other arguments and semantics match
  * those of IOMMU_IOAS_MAP.
+ *
+ * A guest_memfd is also accepted, with @start and @length page aligned. Its
+ * pages are not pinned: only its shared pages that have memory are mapped, a
+ * page as soon as it gets memory, and the mapping follows conversions, punch
+ * holes and its provider's changes. Read-only pages are mapped without write
+ * permission, or not at all without IOMMU_IOAS_MAP_READABLE. Such a mapping
+ * cannot be split by a partial unmap, accessed by the kernel, moved by
+ * IOMMU_IOAS_CHANGE_PROCESS, or used with dirty tracking or the no-IOMMU mode,
+ * and a page at physical address 0 cannot be mapped.
  */
 struct iommu_ioas_map_file {
 	__u32 size;
-- 
2.47.3

[RFC PATCH 10/14] iommufd/selftest: Add mock-domain IOVA queries

[Fred Griffoul]

From: Fred Griffoul <fgriffo@amazon.co.uk>

Tests of guest_memfd mappings cannot know in advance which frame an
IOVA reaches, since a revoke can unmap or replace it, and
IOMMU_TEST_OP_MD_CHECK_MAP only compares a range against a user buffer.

Add IOMMU_TEST_OP_MD_CHECK_MAPPED, which checks that a range of a mock
domain is all mapped or all unmapped, and IOMMU_TEST_OP_MD_IOVA_TO_PHYS,
which returns the physical address behind an IOVA. They hold
domains_rwsem for writing, so that a userspace unmap cannot free a
page-table level during the walk; they are not safe against a
concurrent guest_memfd update or dma-buf revoke, which the selftests do
not do.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Fred Griffoul <fgriffo@amazon.co.uk>
---
 drivers/iommu/iommufd/iommufd_test.h |  18 ++++
 drivers/iommu/iommufd/selftest.c     | 125 +++++++++++++++++++++++++++
 2 files changed, 143 insertions(+)

diff --git a/drivers/iommu/iommufd/iommufd_test.h b/drivers/iommu/iommufd/iommufd_test.h
index 52b78cbcc920..3fffcb2b20d7 100644
--- a/drivers/iommu/iommufd/iommufd_test.h
+++ b/drivers/iommu/iommufd/iommufd_test.h
@@ -31,6 +31,8 @@ enum {
 	IOMMU_TEST_OP_PASID_CHECK_HWPT,
 	IOMMU_TEST_OP_DMABUF_GET,
 	IOMMU_TEST_OP_DMABUF_REVOKE,
+	IOMMU_TEST_OP_MD_CHECK_MAPPED,
+	IOMMU_TEST_OP_MD_IOVA_TO_PHYS,
 };
 
 enum {
@@ -193,6 +195,22 @@ struct iommu_test_cmd {
 			__s32 dmabuf_fd;
 			__u32 revoked;
 		} dmabuf_revoke;
+		struct {
+			/*
+			 * 1: every page in [iova, iova+length) must be mapped;
+			 * 0: none of them may be. Any page in the other state
+			 * fails with -ENOENT. Any other value, or a non-zero
+			 * __reserved, is -EINVAL.
+			 */
+			__u32 mapped;
+			__u32 __reserved;
+			__aligned_u64 iova;
+			__aligned_u64 length;
+		} check_mapped;
+		struct {
+			__aligned_u64 iova;
+			__aligned_u64 out_phys;	/* 0 if unmapped */
+		} iova_to_phys;
 	};
 	__u32 last;
 };
diff --git a/drivers/iommu/iommufd/selftest.c b/drivers/iommu/iommufd/selftest.c
index ee706f18f7e9..378d7c46327e 100644
--- a/drivers/iommu/iommufd/selftest.c
+++ b/drivers/iommu/iommufd/selftest.c
@@ -2068,6 +2068,122 @@ static int iommufd_test_dmabuf_get(struct iommufd_ucmd *ucmd,
 	return rc;
 }
 
+/*
+ * True if [iova, iova + length) lies inside the domain's aperture.  An IOVA
+ * that the domain cannot map would read as unmapped, so the queries refuse
+ * it rather than report it as such.
+ */
+static bool mock_domain_covers(struct mock_iommu_domain *mock,
+			       unsigned long iova, size_t length)
+{
+	struct iommu_domain_geometry *geo = &mock->domain.geometry;
+	unsigned long last;
+
+	if (check_add_overflow(iova, length - 1, &last))
+		return false;
+	return iova >= geo->aperture_start && last <= geo->aperture_end;
+}
+
+/*
+ * iova_to_phys() walks the page table, so it must not run while an unmap
+ * frees a level of it.  A userspace unmap holds the IOAS domains_rwsem for
+ * reading while it unmaps the domains, so holding it for writing keeps such
+ * unmaps away.  A guest_memfd update, e.g. on a conversion or a punch hole,
+ * and a dma-buf revoke unmap under their pages mutex only, so tests must not
+ * run these queries while one is in progress.  A range can cross several
+ * areas and holes, so the queries take one IOAS-wide lock rather than the
+ * pages mutex of each area.
+ */
+static struct rw_semaphore *
+mock_domain_unmap_lock(struct iommufd_hw_pagetable *hwpt)
+{
+	return &to_hwpt_paging(hwpt)->ioas->iopt.domains_rwsem;
+}
+
+/*
+ * Report the physical address the mock domain resolves @iova to, or 0 if
+ * it is unmapped.  Lets a test check that two IOVAs share one frame, or that
+ * an IOVA moved to another frame, without knowing the frames in advance.
+ * Not safe against a concurrent guest_memfd update or dma-buf revoke; see
+ * mock_domain_unmap_lock().
+ */
+static int iommufd_test_md_iova_to_phys(struct iommufd_ucmd *ucmd,
+					unsigned int mockpt_id,
+					unsigned long iova)
+{
+	struct iommu_test_cmd *cmd = ucmd->cmd;
+	struct iommufd_hw_pagetable *hwpt;
+	struct mock_iommu_domain *mock;
+	unsigned int page_size;
+	int rc;
+
+	hwpt = get_md_pagetable(ucmd, mockpt_id, &mock);
+	if (IS_ERR(hwpt))
+		return PTR_ERR(hwpt);
+
+	page_size = 1 << __ffs(mock->domain.pgsize_bitmap);
+	if (iova % page_size || !mock_domain_covers(mock, iova, page_size)) {
+		rc = -EINVAL;
+		goto out_put;
+	}
+	down_write(mock_domain_unmap_lock(hwpt));
+	cmd->iova_to_phys.out_phys =
+		mock->domain.ops->iova_to_phys(&mock->domain, iova);
+	up_write(mock_domain_unmap_lock(hwpt));
+	rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd));
+out_put:
+	iommufd_put_object(ucmd->ictx, &hwpt->obj);
+	return rc;
+}
+
+/*
+ * Check that [@iova, @iova + @length) is fully mapped or fully unmapped.  Not
+ * safe against a concurrent guest_memfd update or dma-buf revoke; see
+ * mock_domain_unmap_lock().
+ */
+static int iommufd_test_md_check_mapped(struct iommufd_ucmd *ucmd,
+					unsigned int mockpt_id,
+					unsigned long iova, size_t length,
+					u32 mapped, u32 reserved)
+{
+	struct iommufd_hw_pagetable *hwpt;
+	struct mock_iommu_domain *mock;
+	unsigned int page_size;
+	int rc = 0;
+
+	if (mapped > 1 || reserved)
+		return -EINVAL;
+
+	hwpt = get_md_pagetable(ucmd, mockpt_id, &mock);
+	if (IS_ERR(hwpt))
+		return PTR_ERR(hwpt);
+
+	page_size = 1 << __ffs(mock->domain.pgsize_bitmap);
+	if (iova % page_size || length % page_size || !length ||
+	    !mock_domain_covers(mock, iova, length)) {
+		rc = -EINVAL;
+		goto out_put;
+	}
+
+	down_write(mock_domain_unmap_lock(hwpt));
+	for (; length; length -= page_size, iova += page_size) {
+		bool is_mapped =
+			mock->domain.ops->iova_to_phys(&mock->domain, iova) != 0;
+
+		if (is_mapped != mapped) {
+			rc = -ENOENT;
+			break;
+		}
+		/* A huge-page mock domain spans 2^38 bytes. */
+		cond_resched();
+	}
+	up_write(mock_domain_unmap_lock(hwpt));
+
+out_put:
+	iommufd_put_object(ucmd->ictx, &hwpt->obj);
+	return rc;
+}
+
 static int iommufd_test_dmabuf_revoke(struct iommufd_ucmd *ucmd, int fd,
 				      bool revoked)
 {
@@ -2180,6 +2296,15 @@ int iommufd_test(struct iommufd_ucmd *ucmd)
 		return iommufd_test_dmabuf_revoke(ucmd,
 						  cmd->dmabuf_revoke.dmabuf_fd,
 						  cmd->dmabuf_revoke.revoked);
+	case IOMMU_TEST_OP_MD_CHECK_MAPPED:
+		return iommufd_test_md_check_mapped(ucmd, cmd->id,
+						    cmd->check_mapped.iova,
+						    cmd->check_mapped.length,
+						    cmd->check_mapped.mapped,
+						    cmd->check_mapped.__reserved);
+	case IOMMU_TEST_OP_MD_IOVA_TO_PHYS:
+		return iommufd_test_md_iova_to_phys(ucmd, cmd->id,
+						    cmd->iova_to_phys.iova);
 	default:
 		return -EOPNOTSUPP;
 	}
-- 
2.47.3

[RFC PATCH 11/14] KVM: guest_memfd: Add kvm_gmem_restore() for PFN mode providers

[Fred Griffoul]

From: Fred Griffoul <fgriffo@amazon.co.uk>

A device cannot fault, so when a PFN mode provider gives frames to pages
that had none, or makes a read-only frame writable, a subsystem that
maps the file must be told. kvm_gmem_revoke() does not fit: it would
zap the guest's and the VMM's mappings and reclaim frames that a guest
may already use.

Add kvm_gmem_restore(), the counterpart to kvm_gmem_revoke() that the
sample provider leaves for users that cannot fault. It only updates the
exports, under the shared invalidate lock.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Fred Griffoul <fgriffo@amazon.co.uk>
---
 arch/x86/kvm/Makefile       |  1 +
 include/linux/guest_memfd.h | 37 ++++++++++++++++++++++---------------
 virt/kvm/guest_memfd.c      | 35 +++++++++++++++++++++++++++++++++--
 3 files changed, 56 insertions(+), 17 deletions(-)

diff --git a/arch/x86/kvm/Makefile b/arch/x86/kvm/Makefile
index 25fb3d6e2032..3255280884b5 100644
--- a/arch/x86/kvm/Makefile
+++ b/arch/x86/kvm/Makefile
@@ -63,6 +63,7 @@ exports_grep_trailer := --include='*.[ch]' -nrw $(srctree)/virt/kvm $(srctree)/a
 				-e kvm_write_track_remove_gfn \
 				-e kvm_file_to_kvm_fn \
 				-e kvm_gmem_revoke \
+				-e kvm_gmem_restore \
 				-e kvm_gmem_export
 
 # Force grep to emit a goofy group separator that can in turn be replaced with
diff --git a/include/linux/guest_memfd.h b/include/linux/guest_memfd.h
index f8e89723dd68..45572d929bd7 100644
--- a/include/linux/guest_memfd.h
+++ b/include/linux/guest_memfd.h
@@ -75,12 +75,18 @@ struct guest_memfd_provider_operations {
 #ifdef CONFIG_KVM_GUEST_MEMFD
 int kvm_gmem_revoke(struct inode *inode, pgoff_t start, size_t nr_pages,
 		    const unsigned long *pfn);
+void kvm_gmem_restore(struct inode *inode, pgoff_t start, size_t nr_pages);
 #else
 static inline int kvm_gmem_revoke(struct inode *inode, pgoff_t start, size_t nr_pages,
 				  const unsigned long *pfn)
 {
 	return -EOPNOTSUPP;
 }
+
+static inline void kvm_gmem_restore(struct inode *inode, pgoff_t start,
+				    size_t nr_pages)
+{
+}
 #endif
 
 /*
@@ -90,14 +96,15 @@ static inline int kvm_gmem_revoke(struct inode *inode, pgoff_t start, size_t nr_
  * nor fault them.  It attaches to the file, looks up the frame behind each
  * shared page without taking a reference, and guest_memfd calls its update()
  * whenever what a lookup returns for a range may have changed: conversions,
- * punch holes, memory failure, a provider's kvm_gmem_revoke(), fallocate() and
- * pages getting their first frame.  update() makes the consumer's mappings of
- * the range match what a lookup returns now: a frame, or a hole for a private
- * page or a page without a frame.  It may be called for pages that did not
- * change, and for ranges that do not follow the blocks that lookups report:
- * the consumer may then remap whole blocks, also outside the range, e.g. to
- * merge them into larger mappings.  A consumer that can fault may instead only
- * remove its mappings of the range.
+ * punch holes, memory failure, a provider's kvm_gmem_revoke() or
+ * kvm_gmem_restore(), fallocate() and pages getting their first frame.
+ * update() makes the consumer's mappings of the range match what a lookup
+ * returns now: a frame, or a hole for a private page or a page without a
+ * frame.  It may be called for pages that did not change, and for ranges that
+ * do not follow the blocks that lookups report: the consumer may then remap
+ * whole blocks, also outside the range, e.g. to merge them into larger
+ * mappings.  A consumer that can fault may instead only remove its mappings of
+ * the range.
  *
  * Until update() returns, the frames it replaces stay valid, so a consumer
  * may switch each mapping to its new target in one step; once it returns,
@@ -106,14 +113,14 @@ static inline int kvm_gmem_revoke(struct inode *inode, pgoff_t start, size_t nr_
  * beforehand.
  *
  * guest_memfd calls update() with its invalidate lock held, exclusive when it
- * takes frames away or pages convert, and shared when pages only gain frames
- * and on memory failure, possibly with a folio of the file locked; so update()
- * may only call kvm_gmem_export_lookup_pfn().  Under the shared lock, updates
- * may run at the same time, also for overlapping ranges.  The consumer holds a
- * lock across a lookup and the mapping it installs, which update() also takes.
- * Lock order:
+ * takes frames or write access away or pages convert, and shared when pages
+ * only gain frames or write access and on memory failure, possibly with a
+ * folio of the file locked; so update() may only call
+ * kvm_gmem_export_lookup_pfn().  Under the shared lock, updates may run at the
+ * same time, also for overlapping ranges.  The consumer holds a lock across a
+ * lookup and the mapping it installs, which update() also takes.  Lock order:
  *
- *   PFN mode provider lock (across kvm_gmem_revoke())
+ *   PFN mode provider lock (across kvm_gmem_revoke() and kvm_gmem_restore())
  *     guest_memfd invalidate lock
  *       consumer lock
  *         provider leaf lock (get_pfn(), lookup_pfn_range())
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 42bb5f263e23..37ff2fae89ae 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -431,8 +431,8 @@ static void kvm_gmem_invalidate_end(struct inode *inode, pgoff_t start,
 
 /*
  * What a lookup returns for [start, end) may have changed: exclusive lock when
- * frames were taken away or pages converted, at least shared when pages only
- * gained frames and on memory failure.
+ * frames or write access were taken away or pages converted, at least shared
+ * when pages only gained frames or write access and on memory failure.
  */
 static void kvm_gmem_update_exports(struct inode *inode, pgoff_t start,
 				    pgoff_t end)
@@ -592,6 +592,37 @@ int kvm_gmem_revoke(struct inode *inode, pgoff_t start, size_t nr_pages,
 }
 EXPORT_SYMBOL_GPL(kvm_gmem_revoke);
 
+/**
+ * kvm_gmem_restore - Tell guest_memfd that memory of a range is now available.
+ * @inode:    The guest_memfd inode.
+ * @start:    Starting page index within the file.
+ * @nr_pages: Number of pages.
+ *
+ * A PFN mode provider calls this when it gives frames to pages that had none,
+ * unless get_pfn() does so for the block it reports, and when it makes a
+ * read-only frame writable.  Nothing is taken away, so the guest and the VMM
+ * need nothing: they fault the pages in, or get write access on their next
+ * write.  The subsystems that map the file cannot, and are told to map the
+ * range again.  A change that takes something away, a frame or write access,
+ * needs kvm_gmem_revoke() instead, which updates them too.  Does nothing past
+ * the end of the file.
+ *
+ * Context: May sleep.  Must not be called with the provider's leaf lock held.
+ */
+void kvm_gmem_restore(struct inode *inode, pgoff_t start, size_t nr_pages)
+{
+	pgoff_t max_pages = i_size_read(inode) >> PAGE_SHIFT;
+
+	if (start >= max_pages)
+		return;
+	nr_pages = min_t(size_t, nr_pages, max_pages - start);
+
+	filemap_invalidate_lock_shared(inode->i_mapping);
+	kvm_gmem_update_exports(inode, start, start + nr_pages);
+	filemap_invalidate_unlock_shared(inode->i_mapping);
+}
+EXPORT_SYMBOL_GPL(kvm_gmem_restore);
+
 /*
  * Zero a folio on its first use.  Lookups find it from then on, with no
  * reference held, so mark it dirty: memory-failure handling drops clean
-- 
2.47.3

[RFC PATCH 12/14] samples/kvm: Add read-only pages and large blocks to the sample provider

[Fred Griffoul]

From: Fred Griffoul <fgriffo@amazon.co.uk>

The sample provider reports every page as writable and of order 0, and
does nothing when a page comes back, so tests cannot exercise read-only
frames, large mappings, or a device mapping that follows a page made
present again.

Add MPS_SET_READONLY, report the largest aligned block of pages in the
same state, and call kvm_gmem_restore() when pages become present or
writable again; making pages read only revokes them.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Fred Griffoul <fgriffo@amazon.co.uk>
---
 samples/kvm/mem_provider_sample.c | 89 +++++++++++++++++++++++++++----
 samples/kvm/mem_provider_sample.h |  9 ++++
 2 files changed, 87 insertions(+), 11 deletions(-)

diff --git a/samples/kvm/mem_provider_sample.c b/samples/kvm/mem_provider_sample.c
index f30654a60dd8..670e36207e59 100644
--- a/samples/kvm/mem_provider_sample.c
+++ b/samples/kvm/mem_provider_sample.c
@@ -6,7 +6,8 @@
  *
  * The control device /dev/mem_provider_sample creates provider files.
  * Each file allocates its own contiguous memory from alloc_contig_pages().
- * Pages can be dynamically marked present or absent.
+ * Pages can be dynamically marked present or absent, and read only (for
+ * tests).
  * See mem_provider_sample.h.
  *
  * Confidential VMs are not supported.
@@ -39,6 +40,7 @@ struct mps_info {
 	struct page *cma_pages;		/* from alloc_contig_pages() */
 
 	unsigned long *absent;		/* taken away by MPS_SET_PRESENT */
+	unsigned long *readonly;	/* set by MPS_SET_READONLY */
 
 	struct inode *gmem_inode;
 };
@@ -83,6 +85,9 @@ static void mps_set_present(struct mps_info *info, unsigned long start,
 	mutex_lock(&info->lock);
 	if (present) {
 		bitmap_clear(info->absent, start, end - start);
+		/* Tell the users of guest_memfd that cannot fault. */
+		if (info->gmem_inode)
+			kvm_gmem_restore(info->gmem_inode, start, end - start);
 	} else {
 		bitmap_set(info->absent, start, end - start);
 		if (info->gmem_inode) {
@@ -95,6 +100,31 @@ static void mps_set_present(struct mps_info *info, unsigned long start,
 	mutex_unlock(&info->lock);
 }
 
+/*
+ * Change what get_pfn() reports for [start, end).  Making it read only takes
+ * write access away, so revoke it: the users of the guest_memfd map it again
+ * read only.  Making it writable takes nothing away, so only tell the users
+ * that cannot fault.  The pages keep their contents.
+ */
+static void mps_set_readonly(struct mps_info *info, unsigned long start,
+			     unsigned long end, bool value)
+{
+	unsigned long pfn = info->base_pfn + start;
+
+	mutex_lock(&info->lock);
+	if (value) {
+		bitmap_set(info->readonly, start, end - start);
+		if (info->gmem_inode)
+			kvm_gmem_revoke(info->gmem_inode, start, end - start,
+					&pfn);
+	} else {
+		bitmap_clear(info->readonly, start, end - start);
+		if (info->gmem_inode)
+			kvm_gmem_restore(info->gmem_inode, start, end - start);
+	}
+	mutex_unlock(&info->lock);
+}
+
 static long mps_get_stats(struct mps_info *info, void __user *uarg)
 {
 	struct mps_stats st = {};
@@ -114,11 +144,13 @@ static int mps_file_release(struct inode *inode, struct file *file)
 	if (info->cma_pages)
 		free_contig_range(info->base_pfn, info->npages);
 	bitmap_free(info->absent);
+	bitmap_free(info->readonly);
 	kfree(info);
 	return 0;
 }
 
-static long mps_file_set_present(struct mps_info *info, void __user *uarg)
+static long mps_file_set_range(struct mps_info *info, void __user *uarg,
+			       unsigned int cmd)
 {
 	struct mps_set_present sp;
 	unsigned long start, end;
@@ -129,9 +161,13 @@ static long mps_file_set_present(struct mps_info *info, void __user *uarg)
 	if (sp.value > 1 || sp.pad)
 		return -EINVAL;
 	ret = mps_range(info, sp.offset, sp.len, &start, &end);
-	if (!ret)
+	if (ret)
+		return ret;
+	if (cmd == MPS_SET_PRESENT)
 		mps_set_present(info, start, end, sp.value);
-	return ret;
+	else
+		mps_set_readonly(info, start, end, sp.value);
+	return 0;
 }
 
 static long mps_file_ioctl(struct file *file, unsigned int cmd,
@@ -144,7 +180,8 @@ static long mps_file_ioctl(struct file *file, unsigned int cmd,
 	case MPS_GET_STATS:
 		return mps_get_stats(info, uarg);
 	case MPS_SET_PRESENT:
-		return mps_file_set_present(info, uarg);
+	case MPS_SET_READONLY:
+		return mps_file_set_range(info, uarg, cmd);
 	default:
 		return -ENOTTY;
 	}
@@ -178,11 +215,28 @@ static void mps_gmem_release(void *provider)
 	fput(resource_file);
 }
 
+/* Is every bit of @bitmap in [first, first + n) equal to @value? */
+static bool mps_bits_are(unsigned long *bitmap, unsigned long first,
+			 unsigned long n, bool value)
+{
+	unsigned long end = first + n;
+
+	return (value ? find_next_zero_bit(bitmap, end, first) :
+			find_next_bit(bitmap, end, first)) >= end;
+}
+
+/*
+ * The frame behind @index, and the largest aligned block around it that is
+ * present and has the same attributes: the file's frames are contiguous.
+ * Lockless: a change is followed by kvm_gmem_revoke() or kvm_gmem_restore().
+ */
 static int mps_gmem_get_pfn(void *provider, pgoff_t index, unsigned long *pfn,
 			    int *max_order, u32 *attrs)
 {
 	struct file *resource_file = provider;
 	struct mps_info *info = resource_file->private_data;
+	bool ro;
+	int order;
 
 	if (index >= info->npages)
 		return -EINVAL;
@@ -190,10 +244,22 @@ static int mps_gmem_get_pfn(void *provider, pgoff_t index, unsigned long *pfn,
 	if (test_bit(index, info->absent))
 		return -EFAULT;
 
+	ro = test_bit(index, info->readonly);
 	*pfn = info->base_pfn + index;
-	if (max_order)
-		*max_order = 0;
-
+	*attrs = ro ? GUEST_MEMFD_PFN_READONLY : 0;
+
+	if (!max_order)
+		return 0;
+	for (order = PUD_ORDER; order; order--) {
+		unsigned long first = ALIGN_DOWN(index, 1UL << order);
+		unsigned long n = 1UL << order;
+
+		if (first + n <= info->npages &&
+		    mps_bits_are(info->absent, first, n, false) &&
+		    mps_bits_are(info->readonly, first, n, ro))
+			break;
+	}
+	*max_order = order;
 	return 0;
 }
 
@@ -255,9 +321,10 @@ static long mps_ctl_new_file(void __user *uarg)
 		return -ENOMEM;
 
 	info->absent = bitmap_zalloc(npages, GFP_KERNEL);
-	if (!info->absent) {
+	info->readonly = bitmap_zalloc(npages, GFP_KERNEL);
+	if (!info->absent || !info->readonly) {
 		ret = -ENOMEM;
-		goto err_free_info;
+		goto err_free_absent;
 	}
 
 	pages = alloc_contig_pages(npages, GFP_KERNEL | __GFP_ZERO,
@@ -293,7 +360,7 @@ static long mps_ctl_new_file(void __user *uarg)
 	free_contig_range(page_to_pfn(pages), npages);
 err_free_absent:
 	bitmap_free(info->absent);
-err_free_info:
+	bitmap_free(info->readonly);
 	kfree(info);
 	return ret;
 }
diff --git a/samples/kvm/mem_provider_sample.h b/samples/kvm/mem_provider_sample.h
index b9f10286cb30..57f494dc46fa 100644
--- a/samples/kvm/mem_provider_sample.h
+++ b/samples/kvm/mem_provider_sample.h
@@ -51,4 +51,13 @@ struct mps_set_present {
 #define MPS_SET_PRESENT \
 	_IOW(MPS_IOCTL_BASE, 12, struct mps_set_present)
 
+/*
+ * ioctl on a provider fd, with the same argument as MPS_SET_PRESENT: report
+ * [@offset, @offset + @len) as read only (GUEST_MEMFD_PFN_READONLY, @value 1)
+ * or writable again (@value 0).  Every mapping follows.  For tests: the
+ * sample does not really share the pages.
+ */
+#define MPS_SET_READONLY \
+	_IOW(MPS_IOCTL_BASE, 13, struct mps_set_present)
+
 #endif /* _SAMPLES_KVM_MEM_PROVIDER_SAMPLE_H */
-- 
2.47.3

[RFC PATCH 13/14] samples/kvm: Back absent pages on demand in the sample provider

[Fred Griffoul]

From: Fred Griffoul <fgriffo@amazon.co.uk>

The sample provider fails get_pfn() for an absent page and keeps its
frames on a punch hole, so tests cannot exercise a provider that backs
a hole when it is asked for a frame, one that backs more than the page
asked for, or one that frees the frames that a punch hole gives back.

Add MPS_NEW_FILE_FILL: get_pfn() then backs an absent page with the
whole 2 MiB block around it, and put_pfn_range() makes the pages absent
again.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Fred Griffoul <fgriffo@amazon.co.uk>
---
 samples/kvm/mem_provider_sample.c | 29 ++++++++++++++++++++++++-----
 samples/kvm/mem_provider_sample.h |  9 ++++++++-
 2 files changed, 32 insertions(+), 6 deletions(-)

diff --git a/samples/kvm/mem_provider_sample.c b/samples/kvm/mem_provider_sample.c
index 670e36207e59..41e0ce5b6bc6 100644
--- a/samples/kvm/mem_provider_sample.c
+++ b/samples/kvm/mem_provider_sample.c
@@ -41,6 +41,7 @@ struct mps_info {
 
 	unsigned long *absent;		/* taken away by MPS_SET_PRESENT */
 	unsigned long *readonly;	/* set by MPS_SET_READONLY */
+	bool fill;			/* MPS_NEW_FILE_FILL */
 
 	struct inode *gmem_inode;
 };
@@ -62,8 +63,15 @@ static void mps_gmem_put_pfn_range(void *provider, pgoff_t index,
 				   size_t nr_pages)
 {
 	struct file *resource_file = provider;
+	struct mps_info *info = resource_file->private_data;
+	size_t i;
 
-	mps_zero_range(resource_file->private_data, index, nr_pages);
+	mps_zero_range(info, index, nr_pages);
+	if (!info->fill || index >= info->npages)
+		return;
+	/* Lockless, like get_pfn(): guest_memfd holds its invalidate lock. */
+	for (i = index; i < min_t(size_t, index + nr_pages, info->npages); i++)
+		set_bit(i, info->absent);
 }
 
 /* Byte range [offset, offset + len) of @info as page indices. */
@@ -228,7 +236,9 @@ static bool mps_bits_are(unsigned long *bitmap, unsigned long first,
 /*
  * The frame behind @index, and the largest aligned block around it that is
  * present and has the same attributes: the file's frames are contiguous.
- * Lockless: a change is followed by kvm_gmem_revoke() or kvm_gmem_restore().
+ * Lockless: a change is followed by kvm_gmem_revoke() or kvm_gmem_restore(),
+ * except a fill, which only gives frames to holes; guest_memfd tells its users
+ * about the block.
  */
 static int mps_gmem_get_pfn(void *provider, pgoff_t index, unsigned long *pfn,
 			    int *max_order, u32 *attrs)
@@ -241,8 +251,16 @@ static int mps_gmem_get_pfn(void *provider, pgoff_t index, unsigned long *pfn,
 	if (index >= info->npages)
 		return -EINVAL;
 
-	if (test_bit(index, info->absent))
-		return -EFAULT;
+	if (test_bit(index, info->absent)) {
+		unsigned long first = ALIGN_DOWN(index, PTRS_PER_PMD);
+		unsigned long i;
+
+		if (!info->fill)
+			return -EFAULT;
+		/* Absent pages are zeroed already. */
+		for (i = first; i < min(first + PTRS_PER_PMD, info->npages); i++)
+			clear_bit(i, info->absent);
+	}
 
 	ro = test_bit(index, info->readonly);
 	*pfn = info->base_pfn + index;
@@ -307,7 +325,7 @@ static long mps_ctl_new_file(void __user *uarg)
 
 	if (copy_from_user(&nf, uarg, sizeof(nf)))
 		return -EFAULT;
-	if (nf.flags || nf.pad)
+	if ((nf.flags & ~MPS_NEW_FILE_FILL) || nf.pad)
 		return -EINVAL;
 	if (!nf.size || !PAGE_ALIGNED(nf.size))
 		return -EINVAL;
@@ -337,6 +355,7 @@ static long mps_ctl_new_file(void __user *uarg)
 	info->cma_pages = pages;
 	info->base_pfn = page_to_pfn(pages);
 	info->npages = npages;
+	info->fill = nf.flags & MPS_NEW_FILE_FILL;
 	mutex_init(&info->lock);
 
 	fd = get_unused_fd_flags(O_CLOEXEC);
diff --git a/samples/kvm/mem_provider_sample.h b/samples/kvm/mem_provider_sample.h
index 57f494dc46fa..45ec9605b593 100644
--- a/samples/kvm/mem_provider_sample.h
+++ b/samples/kvm/mem_provider_sample.h
@@ -19,11 +19,18 @@
  * Returns the provider fd.
  */
 struct mps_new_file {
-	__u32 flags;	/* Reserved, must be 0 */
+	__u32 flags;	/* MPS_NEW_FILE_* */
 	__u32 pad;
 	__u64 size;	/* bytes, page aligned */
 };
 
+/*
+ * get_pfn() backs an absent page, with every page of the 2 MiB block around
+ * it, instead of failing, and put_pfn_range() takes the pages back: a
+ * provider that allocates on demand and frees on a punch hole, for tests.
+ */
+#define MPS_NEW_FILE_FILL	(1U << 0)
+
 #define MPS_NEW_FILE \
 	_IOW(MPS_IOCTL_BASE, 5, struct mps_new_file)
 
-- 
2.47.3

[RFC PATCH 14/14] KVM: selftests: Test guest_memfd mapped by iommufd

[Fred Griffoul]

From: Fred Griffoul <fgriffo@amazon.co.uk>

Nothing tests that a device mapping of a guest_memfd follows the file.

Add gmem_conversion_iommufd_test. A KVM_X86_SW_PROTECTED_VM maps its
guest_memfd into an iommufd mock domain, on native folios, a tmpfs
resource and the sample provider in PFN mode, and the test checks what
the guest, the VMM and the device see after each change: conversions,
allocations, punch holes, memory failure, and the provider's revokes and
restores. It is skipped without CONFIG_IOMMUFD_TEST, the PFN mode cases
without the sample, and memory failure without CONFIG_MEMORY_FAILURE.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Fred Griffoul <fgriffo@amazon.co.uk>
---
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../kvm/x86/gmem_conversion_iommufd_test.c    | 956 ++++++++++++++++++
 2 files changed, 957 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/x86/gmem_conversion_iommufd_test.c

diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm
index 5f7118bddfc3..4520c102b695 100644
--- a/tools/testing/selftests/kvm/Makefile.kvm
+++ b/tools/testing/selftests/kvm/Makefile.kvm
@@ -162,6 +162,7 @@ TEST_GEN_PROGS_x86 += x86/triple_fault_event_test
 TEST_GEN_PROGS_x86 += x86/recalc_apic_map_test
 TEST_GEN_PROGS_x86 += x86/aperfmperf_test
 TEST_GEN_PROGS_x86 += x86/guest_memfd_conversions_test
+TEST_GEN_PROGS_x86 += x86/gmem_conversion_iommufd_test
 TEST_GEN_PROGS_x86 += access_tracking_perf_test
 TEST_GEN_PROGS_x86 += coalesced_io_test
 TEST_GEN_PROGS_x86 += dirty_log_perf_test
diff --git a/tools/testing/selftests/kvm/x86/gmem_conversion_iommufd_test.c b/tools/testing/selftests/kvm/x86/gmem_conversion_iommufd_test.c
new file mode 100644
index 000000000000..5d17b0e2dd43
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/gmem_conversion_iommufd_test.c
@@ -0,0 +1,956 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * gmem_conversion_iommufd_test - guest_memfd mapped by iommufd.
+ *
+ * A software protected VM has its guest_memfd mapped into an iommufd mock
+ * domain, and converts pages between shared and private.  A conversion to
+ * private must take the page away from the device before the guest can use
+ * it as private; a conversion to shared gives the device the frame again.
+ * The same scenario runs on each backing of a guest_memfd:
+ *
+ *  - native folios;
+ *  - folios from a tmpfs resource (GUEST_MEMFD_FLAG_USE_RESOURCE on a mount);
+ *  - frames of a PFN mode provider (a file of the sample provider).
+ *
+ * The folio backings also check that the device gets the pages that
+ * fallocate() or a first host access allocates, whether the guest_memfd is
+ * mapped before or after, that it loses a page that memory failure poisons,
+ * and that two IOAS mapping one guest_memfd both follow it.  The PFN mode
+ * provider then gets its own checks: read-only frames, which the guest and the
+ * VMM cannot write and the device keeps mapped, private frames, read-only
+ * frames as private holes, absent frames made private, the provider taking
+ * back a private page and giving it back, a punch hole, a 2 MiB private
+ * mapping, and a provider that backs a whole 2 MiB block when one page of it
+ * is asked for and frees it on a punch hole.
+ *
+ * Requires CONFIG_IOMMUFD_TEST; the PFN mode scenarios also require
+ * samples/kvm/mem_provider_sample.ko.
+ */
+#include <fcntl.h>
+#include <errno.h>
+#include <setjmp.h>
+#include <signal.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <string.h>
+#include <unistd.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+
+#include <linux/iommufd.h>
+
+#include "test_util.h"
+#include "kvm_util.h"
+#include "processor.h"
+
+/*
+ * The iommufd mock domain test interface.  The iommufd selftest helpers
+ * bring their own harness, which does not mix with the KVM selftest library.
+ */
+#include "../../../../../drivers/iommu/iommufd/iommufd_test.h"
+#include "../../../../../samples/kvm/mem_provider_sample.h"
+
+#define PAGE		0x1000ULL
+#define HPAGE		0x200000ULL		/* 2 MiB */
+#define FILE_SIZE	0x400000ULL		/* 4 MiB per guest_memfd */
+#define GPA		(1ULL << 32)		/* the VM maps its file here */
+#define IOVA		(1ULL << 28)		/* inside the mock aperture */
+
+struct guest_args {
+	uint64_t write_gpa;	/* 0 to skip */
+	uint64_t write_val;
+	uint64_t read_gpa;	/* 0 to skip; the value goes to GUEST_SYNC */
+};
+
+static __maybe_unused void guest_code(struct guest_args *a)
+{
+	if (a->write_gpa)
+		*(volatile uint64_t *)a->write_gpa = a->write_val;
+	if (a->read_gpa)
+		GUEST_SYNC(*(volatile uint64_t *)a->read_gpa);
+	GUEST_DONE();
+}
+
+static int ctl = -1;	/* /dev/mem_provider_sample */
+
+static int mps_new_file(uint64_t size)
+{
+	struct mps_new_file nf = { .size = size };
+	int fd = ioctl(ctl, MPS_NEW_FILE, &nf);
+
+	TEST_ASSERT(fd >= 0, "MPS_NEW_FILE(%#llx) errno=%d",
+		    (unsigned long long)size, errno);
+	return fd;
+}
+
+/* MPS_SET_PRESENT or MPS_SET_READONLY on a range. */
+static void mps_set(int fd, unsigned long cmd, uint64_t off, uint64_t len,
+		    bool value)
+{
+	struct mps_set_present sp = { .value = value, .offset = off,
+				      .len = len };
+
+	TEST_ASSERT(!ioctl(fd, cmd, &sp), "MPS ioctl %#lx errno=%d", cmd,
+		    errno);
+}
+
+struct vmm {
+	const char *name;
+	int child;		/* the resource file, or -1 */
+	bool pfn;		/* the resource is a provider file (PFN mode) */
+	int gmem;		/* the guest_memfd backed by it */
+	int iommufd;
+	uint32_t ioas, stdev, hwpt;
+	struct kvm_vm *vm;
+	struct kvm_vcpu *vcpu;
+	void *hva;		/* host mapping of the whole guest_memfd */
+	gva_t args_gva;
+};
+
+static void vmm_create_type(struct vmm *v, unsigned long type)
+{
+	const struct vm_shape shape = {
+		.mode = VM_MODE_DEFAULT,
+		.type = type,
+	};
+
+	v->vm = __vm_create_shape_with_one_vcpu(shape, &v->vcpu, 0, guest_code);
+	v->args_gva = vm_alloc_page(v->vm);
+}
+
+static bool vmm_iova_mapped(struct vmm *v, uint64_t iova, uint64_t len,
+			    bool mapped)
+{
+	struct iommu_test_cmd cmd = {
+		.size = sizeof(cmd), .op = IOMMU_TEST_OP_MD_CHECK_MAPPED,
+		.id = v->hwpt,
+		.check_mapped = { .mapped = mapped, .iova = iova,
+				  .length = len },
+	};
+
+	return !ioctl(v->iommufd, IOMMU_TEST_CMD, &cmd);
+}
+
+static uint64_t vmm_iova_phys(struct vmm *v, uint64_t iova)
+{
+	struct iommu_test_cmd cmd = {
+		.size = sizeof(cmd), .op = IOMMU_TEST_OP_MD_IOVA_TO_PHYS,
+		.id = v->hwpt,
+		.iova_to_phys = { .iova = iova },
+	};
+
+	if (ioctl(v->iommufd, IOMMU_TEST_CMD, &cmd))
+		return 0;
+	return cmd.iova_to_phys.out_phys;
+}
+
+/*
+ * Map the guest_memfd on a 2 MiB boundary: KVM uses large pages only if the
+ * memslot's guest and host addresses are aligned alike.
+ */
+static void *vmm_map_gmem(struct vmm *v)
+{
+	uint8_t *area, *aligned;
+
+	area = mmap(NULL, FILE_SIZE + HPAGE, PROT_NONE,
+		    MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+	TEST_ASSERT(area != MAP_FAILED, "%s: mmap(reserve) errno=%d", v->name,
+		    errno);
+	aligned = (uint8_t *)(((uintptr_t)area + HPAGE - 1) & ~(HPAGE - 1));
+	if (aligned > area)
+		munmap(area, aligned - area);
+	munmap(aligned + FILE_SIZE, area + HPAGE - aligned);
+
+	return mmap(aligned, FILE_SIZE, PROT_READ | PROT_WRITE,
+		    MAP_SHARED | MAP_FIXED, v->gmem, 0);
+}
+
+/*
+ * Give @resource_fd (a provider file of the sample, a tmpfs mount, or -1 for
+ * native folios) to KVM as the guest_memfd's resource, and the guest_memfd to
+ * iommufd.
+ */
+static void vmm_attach(struct vmm *v, int resource_fd)
+{
+	struct iommu_ioas_alloc alloc = { .size = sizeof(alloc) };
+	struct iommu_test_cmd mock = {
+		.size = sizeof(mock), .op = IOMMU_TEST_OP_MOCK_DOMAIN,
+	};
+	struct iommu_ioas_map_file map = {
+		.size = sizeof(map),
+		.flags = IOMMU_IOAS_MAP_FIXED_IOVA | IOMMU_IOAS_MAP_READABLE |
+			 IOMMU_IOAS_MAP_WRITEABLE,
+		.start = 0, .length = FILE_SIZE, .iova = IOVA,
+	};
+	int r;
+
+	uint64_t flags = GUEST_MEMFD_FLAG_MMAP | GUEST_MEMFD_FLAG_INIT_SHARED;
+
+	v->child = resource_fd;
+	if (v->pfn)
+		flags |= GUEST_MEMFD_FLAG_PFN_MODE;
+
+	/* KVM: a guest_memfd on the resource, behind one memslot. */
+	if (resource_fd >= 0)
+		v->gmem = vm_create_guest_memfd_resource(v->vm, FILE_SIZE,
+							 flags | GUEST_MEMFD_FLAG_USE_RESOURCE,
+							 resource_fd);
+	else
+		v->gmem = vm_create_guest_memfd(v->vm, FILE_SIZE, flags);
+	v->hva = vmm_map_gmem(v);
+	TEST_ASSERT(v->hva != MAP_FAILED, "%s: mmap(guest_memfd) errno=%d",
+		    v->name, errno);
+	r = __vm_set_user_memory_region2(v->vm, 10, KVM_MEM_GUEST_MEMFD, GPA,
+					 FILE_SIZE, v->hva, v->gmem, 0);
+	TEST_ASSERT(!r, "%s: SET_USER_MEMORY_REGION2 errno=%d", v->name,
+		    errno);
+	virt_map(v->vm, GPA, GPA, FILE_SIZE / PAGE);
+
+	/* iommufd: the guest_memfd, in an IOAS on a mock domain. */
+	v->iommufd = open("/dev/iommu", O_RDWR);
+	__TEST_REQUIRE(v->iommufd >= 0, "iommufd unavailable");
+	TEST_ASSERT(!ioctl(v->iommufd, IOMMU_IOAS_ALLOC, &alloc),
+		    "IOAS_ALLOC errno=%d", errno);
+	v->ioas = alloc.out_ioas_id;
+
+	mock.id = v->ioas;
+	__TEST_REQUIRE(!ioctl(v->iommufd, IOMMU_TEST_CMD, &mock),
+		       "no iommufd mock domain (CONFIG_IOMMUFD_TEST?)");
+	v->stdev = mock.mock_domain.out_stdev_id;
+	v->hwpt = mock.mock_domain.out_hwpt_id;
+
+	map.ioas_id = v->ioas;
+	map.fd = v->gmem;
+	TEST_ASSERT(!ioctl(v->iommufd, IOMMU_IOAS_MAP_FILE, &map),
+		    "%s: IOAS_MAP_FILE(guest_memfd) errno=%d", v->name, errno);
+}
+
+/*
+ * Continue the guest.  KVM_RUN fails with EFAULT for KVM_EXIT_MEMORY_FAULT,
+ * which several scenarios expect; the expect_*() helpers check the exit.
+ */
+static void vmm_resume(struct vmm *v)
+{
+	int r = _vcpu_run(v->vcpu);
+
+	TEST_ASSERT(!r || (errno == EFAULT &&
+			   v->vcpu->run->exit_reason == KVM_EXIT_MEMORY_FAULT),
+		    "%s: KVM_RUN r=%d errno=%d exit=%s", v->name, r, errno,
+		    exit_reason_str(v->vcpu->run->exit_reason));
+}
+
+static void vmm_run(struct vmm *v, uint64_t write_gpa, uint64_t write_val,
+		    uint64_t read_gpa)
+{
+	struct guest_args *a = addr_gva2hva(v->vm, v->args_gva);
+
+	a->write_gpa = write_gpa;
+	a->write_val = write_val;
+	a->read_gpa = read_gpa;
+	vcpu_arch_set_entry_point(v->vcpu, guest_code);
+	vcpu_args_set(v->vcpu, 1, v->args_gva);
+	vmm_resume(v);
+}
+
+static void expect_done(struct vmm *v)
+{
+	struct kvm_run *run = v->vcpu->run;
+	struct ucall uc;
+
+	TEST_ASSERT(run->exit_reason != KVM_EXIT_MEMORY_FAULT,
+		    "%s: unexpected memory fault at gpa %#llx", v->name,
+		    (unsigned long long)run->memory_fault.gpa);
+	switch (get_ucall(v->vcpu, &uc)) {
+	case UCALL_DONE:
+		return;
+	case UCALL_ABORT:
+		REPORT_GUEST_ASSERT(uc);
+	default:
+		TEST_FAIL("%s: guest exit %s, ucall %lu", v->name,
+			  exit_reason_str(run->exit_reason), uc.cmd);
+	}
+}
+
+static uint64_t expect_sync_then_done(struct vmm *v)
+{
+	struct ucall uc;
+	uint64_t val;
+
+	TEST_ASSERT(get_ucall(v->vcpu, &uc) == UCALL_SYNC, "%s: guest exit %s",
+		    v->name, exit_reason_str(v->vcpu->run->exit_reason));
+	val = uc.args[1];
+	vmm_resume(v);
+	expect_done(v);
+	return val;
+}
+
+static void expect_memory_fault(struct vmm *v, uint64_t gpa)
+{
+	struct kvm_run *run = v->vcpu->run;
+
+	TEST_ASSERT(run->exit_reason == KVM_EXIT_MEMORY_FAULT,
+		    "%s: want KVM_EXIT_MEMORY_FAULT, got %s", v->name,
+		    exit_reason_str(run->exit_reason));
+	TEST_ASSERT(run->memory_fault.gpa == gpa,
+		    "%s: fault at gpa %#llx, want %#llx", v->name,
+		    (unsigned long long)run->memory_fault.gpa,
+		    (unsigned long long)gpa);
+}
+
+static void vmm_destroy(struct vmm *v)
+{
+	close(v->iommufd);
+	kvm_vm_free(v->vm);
+	munmap(v->hva, FILE_SIZE);
+	close(v->gmem);
+	if (v->child >= 0)
+		close(v->child);
+}
+
+static sigjmp_buf host_jmp;
+
+static void host_sig(int sig)
+{
+	siglongjmp(host_jmp, sig);
+}
+
+/*
+ * Read @addr, or write @val to it, through a host mapping.  Return the
+ * signal that the access raised, or 0.
+ */
+static int host_access(volatile uint64_t *addr, bool write, uint64_t val)
+{
+	struct sigaction sa = { .sa_handler = host_sig }, old_bus, old_segv;
+	int sig;
+
+	sigaction(SIGBUS, &sa, &old_bus);
+	sigaction(SIGSEGV, &sa, &old_segv);
+	sig = sigsetjmp(host_jmp, 1);
+	if (!sig) {
+		if (write)
+			*addr = val;
+		else
+			(void)*addr;
+	}
+	sigaction(SIGBUS, &old_bus, NULL);
+	sigaction(SIGSEGV, &old_segv, NULL);
+	return sig;
+}
+
+/*
+ * Poison the page at @addr with MADV_HWPOISON.  Memory failure kills the task
+ * that maps a dirty page with SIGBUS, which counts as done.  Returns 0, or
+ * the errno of madvise().
+ */
+static int host_poison(void *addr)
+{
+	struct sigaction sa = { .sa_handler = host_sig }, old_bus;
+	int r = 0;
+
+	sigaction(SIGBUS, &sa, &old_bus);
+	if (!sigsetjmp(host_jmp, 1) && madvise(addr, PAGE, MADV_HWPOISON))
+		r = errno;
+	sigaction(SIGBUS, &old_bus, NULL);
+	return r;
+}
+
+#define host_read_faults(p)	(host_access(p, false, 0) == SIGBUS)
+#define host_write_faults(p, v)	(host_access(p, true, v) == SIGBUS)
+
+#define MAGIC		0xc0c0a000c0c0a000ULL
+
+/* The page that is converted, inside the file. */
+#define CONV_OFF	(16 * PAGE)
+#define CONV_GPA	(GPA + CONV_OFF)
+#define CONV_IOVA	(IOVA + CONV_OFF)
+
+/* Pages that only fallocate() or the host touch first. */
+#define FALLOC_OFF	(32 * PAGE)
+#define FALLOC2_OFF	(33 * PAGE)
+#define HOST_OFF	(34 * PAGE)
+
+/* A page that memory failure poisons, and its neighbour. */
+#define POISON_OFF	(80 * PAGE)
+#define POISON_IOVA	(IOVA + POISON_OFF)
+
+/* Pages of the file that the PFN mode checks use. */
+#define RO_OFF		(48 * PAGE)
+#define RO_GPA		(GPA + RO_OFF)
+#define PFN_OFF		(64 * PAGE)
+#define PFN_GPA		(GPA + PFN_OFF)
+#define PFN_IOVA	(IOVA + PFN_OFF)
+
+/* The second 2 MiB block of the file, for the large mapping. */
+#define LARGE_OFF	HPAGE
+#define LARGE_GPA	(GPA + LARGE_OFF)
+#define LARGE_IOVA	(IOVA + LARGE_OFF)
+
+enum backing { BACKING_FOLIO, BACKING_TMPFS, BACKING_PFN, NR_BACKINGS };
+
+static const char *const backing_names[] = { "folio", "tmpfs", "pfn" };
+
+/* A software protected VM on @backing, mapped into a mock domain. */
+static void vmm_setup(struct vmm *v, enum backing backing)
+{
+	int resource;
+
+	vmm_create_type(v, KVM_X86_SW_PROTECTED_VM);
+	switch (backing) {
+	case BACKING_FOLIO:
+		resource = -1;
+		break;
+	case BACKING_TMPFS:
+		resource = create_tmpfs_pool_fd("never", true, FILE_SIZE);
+		break;
+	default:
+		resource = mps_new_file(FILE_SIZE);
+		v->pfn = true;
+		break;
+	}
+	vmm_attach(v, resource);
+}
+
+/*
+ * Convert a page to private and back, and check the device, the host and
+ * the guest at each step.  The host and the device reach the page first, so
+ * that a host mapping or an IOPTE left behind would show.
+ */
+static void scenario_convert(enum backing backing)
+{
+	struct vmm v = { .name = backing_names[backing] };
+	volatile uint64_t *host;
+	uint64_t frame, err = 0;
+
+	pr_info("%d. conversions on %s\n", backing + 1, backing_names[backing]);
+	vmm_setup(&v, backing);
+	host = (volatile uint64_t *)(v.hva + CONV_OFF);
+
+	/* Shared: the guest writes, the host and the device see the page. */
+	vmm_run(&v, CONV_GPA, MAGIC, 0);
+	expect_done(&v);
+	TEST_ASSERT(*host == MAGIC, "%s: host does not see the guest's write",
+		    v.name);
+	frame = vmm_iova_phys(&v, CONV_IOVA);
+	TEST_ASSERT(frame, "%s: device does not map the page", v.name);
+
+	/* Private: the device and the host lose it; the guest keeps it. */
+	TEST_ASSERT(!__gmem_set_private(v.gmem, CONV_OFF, PAGE, &err),
+		    "%s: conversion to private errno=%d", v.name, errno);
+	TEST_ASSERT(!vmm_iova_phys(&v, CONV_IOVA),
+		    "%s: device still maps the private page", v.name);
+	TEST_ASSERT(host_read_faults(host),
+		    "%s: host mapping of a private page readable", v.name);
+	vmm_run(&v, 0, 0, CONV_GPA);
+	TEST_ASSERT(expect_sync_then_done(&v) == MAGIC,
+		    "%s: the private page lost the guest's write", v.name);
+	TEST_ASSERT(!vmm_iova_phys(&v, CONV_IOVA),
+		    "%s: device maps the page the guest uses as private",
+		    v.name);
+
+	/* Shared again: the same frame for everyone, contents kept. */
+	TEST_ASSERT(!__gmem_set_shared(v.gmem, CONV_OFF, PAGE, &err),
+		    "%s: conversion to shared errno=%d", v.name, errno);
+	TEST_ASSERT(vmm_iova_phys(&v, CONV_IOVA) == frame,
+		    "%s: device does not map the frame again", v.name);
+	TEST_ASSERT(*host == MAGIC, "%s: host does not see the page again",
+		    v.name);
+	vmm_run(&v, 0, 0, CONV_GPA);
+	TEST_ASSERT(expect_sync_then_done(&v) == MAGIC,
+		    "%s: the guest lost the page on the way back", v.name);
+
+	/* Folios: a punch hole takes the page from the device too. */
+	if (backing != BACKING_PFN) {
+		TEST_ASSERT(!fallocate(v.gmem,
+				       FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE,
+				       CONV_OFF, PAGE),
+			    "%s: punch hole errno=%d", v.name, errno);
+		TEST_ASSERT(!vmm_iova_phys(&v, CONV_IOVA),
+			    "%s: device maps a punched page", v.name);
+		/* Touching it allocates a new page, which the device gets. */
+		vmm_run(&v, CONV_GPA, MAGIC, 0);
+		expect_done(&v);
+		TEST_ASSERT(vmm_iova_phys(&v, CONV_IOVA),
+			    "%s: device does not map the new page", v.name);
+	}
+
+	vmm_destroy(&v);
+}
+
+static void ioas_map_gmem(struct vmm *v)
+{
+	struct iommu_ioas_map_file map = {
+		.size = sizeof(map),
+		.flags = IOMMU_IOAS_MAP_FIXED_IOVA | IOMMU_IOAS_MAP_READABLE |
+			 IOMMU_IOAS_MAP_WRITEABLE,
+		.ioas_id = v->ioas, .fd = v->gmem,
+		.start = 0, .length = FILE_SIZE, .iova = IOVA,
+	};
+
+	TEST_ASSERT(!ioctl(v->iommufd, IOMMU_IOAS_MAP_FILE, &map),
+		    "%s: IOAS_MAP_FILE errno=%d", v->name, errno);
+}
+
+static void ioas_unmap_gmem(struct vmm *v)
+{
+	struct iommu_ioas_unmap unmap = {
+		.size = sizeof(unmap), .ioas_id = v->ioas,
+		.iova = IOVA, .length = FILE_SIZE,
+	};
+
+	TEST_ASSERT(!ioctl(v->iommufd, IOMMU_IOAS_UNMAP, &unmap),
+		    "%s: IOAS_UNMAP errno=%d", v->name, errno);
+}
+
+/*
+ * Folios: the device gets a page as soon as it exists, whichever path
+ * allocates it, and whether the guest_memfd was mapped before or after.
+ */
+static void scenario_alloc(enum backing backing)
+{
+	struct vmm v = { .name = backing_names[backing] };
+	volatile uint64_t *host;
+
+	pr_info("   allocations on %s\n", backing_names[backing]);
+	vmm_setup(&v, backing);
+
+	/* Mapped first: fallocate() allocates the page. */
+	TEST_ASSERT(!vmm_iova_phys(&v, IOVA + FALLOC_OFF),
+		    "%s: device maps a page that does not exist", v.name);
+	TEST_ASSERT(!fallocate(v.gmem, FALLOC_FL_KEEP_SIZE, FALLOC_OFF, PAGE),
+		    "%s: fallocate errno=%d", v.name, errno);
+	TEST_ASSERT(vmm_iova_phys(&v, IOVA + FALLOC_OFF),
+		    "%s: device does not map a page fallocate() allocated",
+		    v.name);
+
+	/* Mapped first: the host's first access allocates the page. */
+	host = (volatile uint64_t *)(v.hva + HOST_OFF);
+	TEST_ASSERT(!vmm_iova_phys(&v, IOVA + HOST_OFF),
+		    "%s: device maps a page the host never touched", v.name);
+	*host = MAGIC;
+	TEST_ASSERT(vmm_iova_phys(&v, IOVA + HOST_OFF),
+		    "%s: device does not map a page the host allocated", v.name);
+	vmm_run(&v, 0, 0, GPA + HOST_OFF);
+	TEST_ASSERT(expect_sync_then_done(&v) == MAGIC,
+		    "%s: guest does not see the host's write", v.name);
+
+	/* Allocated first, while nothing maps the file, then mapped. */
+	ioas_unmap_gmem(&v);
+	TEST_ASSERT(!fallocate(v.gmem, FALLOC_FL_KEEP_SIZE, FALLOC2_OFF, PAGE),
+		    "%s: fallocate errno=%d", v.name, errno);
+	ioas_map_gmem(&v);
+	TEST_ASSERT(vmm_iova_phys(&v, IOVA + FALLOC2_OFF),
+		    "%s: device does not map a page allocated before the map",
+		    v.name);
+	TEST_ASSERT(vmm_iova_phys(&v, IOVA + HOST_OFF),
+		    "%s: device lost a page across the remap", v.name);
+
+	vmm_destroy(&v);
+}
+
+/* An IOAS without a domain: nothing maps an area until a device attaches. */
+static void ioas_check_map_overflow(void)
+{
+	struct vmm v = { .name = "folio" };
+	struct iommu_ioas_alloc alloc = { .size = sizeof(alloc) };
+	struct iommu_ioas_map_file map = {
+		.size = sizeof(map),
+		.flags = IOMMU_IOAS_MAP_FIXED_IOVA | IOMMU_IOAS_MAP_READABLE,
+		.start = -PAGE, .length = PAGE, .iova = IOVA,
+	};
+
+	pr_info("   a guest_memfd range that wraps\n");
+	vmm_setup(&v, BACKING_FOLIO);
+	TEST_ASSERT(!ioctl(v.iommufd, IOMMU_IOAS_ALLOC, &alloc),
+		    "IOAS_ALLOC errno=%d", errno);
+	map.ioas_id = alloc.out_ioas_id;
+	map.fd = v.gmem;
+	TEST_ASSERT(ioctl(v.iommufd, IOMMU_IOAS_MAP_FILE, &map) && errno == EINVAL,
+		    "IOAS_MAP_FILE of a wrapping range: errno=%d, want EINVAL",
+		    errno);
+	vmm_destroy(&v);
+}
+
+/*
+ * Memory failure on a page the device maps: the device loses it, and the
+ * poisoned folio stays in the file, so the host cannot fault in a new one.
+ */
+static void scenario_poison(enum backing backing)
+{
+	struct vmm v = { .name = backing_names[backing] };
+	volatile uint64_t *host, *next;
+	int r;
+
+	pr_info("   memory failure on %s\n", backing_names[backing]);
+	vmm_setup(&v, backing);
+	host = (volatile uint64_t *)(v.hva + POISON_OFF);
+	next = (volatile uint64_t *)(v.hva + POISON_OFF + PAGE);
+	*host = MAGIC;
+	*next = MAGIC;
+	TEST_ASSERT(vmm_iova_phys(&v, POISON_IOVA),
+		    "%s: device does not map the page", v.name);
+
+	r = host_poison((void *)host);
+	if (r == EINVAL || r == EPERM) {
+		pr_info("   skipped: no MADV_HWPOISON (errno %d)\n", r);
+		vmm_destroy(&v);
+		return;
+	}
+	TEST_ASSERT(!r, "%s: MADV_HWPOISON errno=%d", v.name, r);
+
+	TEST_ASSERT(!vmm_iova_phys(&v, POISON_IOVA),
+		    "%s: device maps a poisoned page", v.name);
+	TEST_ASSERT(host_read_faults(host),
+		    "%s: host faulted in a new page over a poisoned one",
+		    v.name);
+	TEST_ASSERT(!vmm_iova_phys(&v, POISON_IOVA),
+		    "%s: device maps a poisoned page after a host fault",
+		    v.name);
+	TEST_ASSERT(vmm_iova_phys(&v, POISON_IOVA + PAGE) && *next == MAGIC,
+		    "%s: the next page was lost", v.name);
+
+	vmm_destroy(&v);
+}
+
+/* A second IOAS on the same guest_memfd, with its own mock domain. */
+static void ioas_attach_second(struct vmm *v, struct vmm *w)
+{
+	struct iommu_ioas_alloc alloc = { .size = sizeof(alloc) };
+	struct iommu_test_cmd mock = {
+		.size = sizeof(mock), .op = IOMMU_TEST_OP_MOCK_DOMAIN,
+	};
+
+	*w = *v;
+	w->name = "second IOAS";
+	TEST_ASSERT(!ioctl(v->iommufd, IOMMU_IOAS_ALLOC, &alloc),
+		    "IOAS_ALLOC errno=%d", errno);
+	w->ioas = alloc.out_ioas_id;
+	mock.id = w->ioas;
+	TEST_ASSERT(!ioctl(v->iommufd, IOMMU_TEST_CMD, &mock),
+		    "MOCK_DOMAIN errno=%d", errno);
+	w->stdev = mock.mock_domain.out_stdev_id;
+	w->hwpt = mock.mock_domain.out_hwpt_id;
+	ioas_map_gmem(w);
+}
+
+/*
+ * Two IOAS map one guest_memfd: both follow its conversions, and either can
+ * go away while the other keeps following.
+ */
+static void scenario_two_ioas(void)
+{
+	struct vmm v = { .name = "folio" }, w;
+	volatile uint64_t *host;
+	uint64_t frame, err = 0;
+
+	pr_info("   two IOAS mapping one guest_memfd\n");
+	vmm_setup(&v, BACKING_FOLIO);
+	ioas_attach_second(&v, &w);
+	host = (volatile uint64_t *)(v.hva + CONV_OFF);
+	*host = MAGIC;
+	frame = vmm_iova_phys(&v, CONV_IOVA);
+	TEST_ASSERT(frame && vmm_iova_phys(&w, CONV_IOVA) == frame,
+		    "the two IOAS do not map the same frame");
+
+	TEST_ASSERT(!__gmem_set_private(v.gmem, CONV_OFF, PAGE, &err),
+		    "conversion to private errno=%d", errno);
+	TEST_ASSERT(!vmm_iova_phys(&v, CONV_IOVA) && !vmm_iova_phys(&w, CONV_IOVA),
+		    "an IOAS maps a private page");
+	TEST_ASSERT(!__gmem_set_shared(v.gmem, CONV_OFF, PAGE, &err),
+		    "conversion to shared errno=%d", errno);
+	TEST_ASSERT(vmm_iova_phys(&v, CONV_IOVA) == frame &&
+		    vmm_iova_phys(&w, CONV_IOVA) == frame,
+		    "an IOAS does not map the page shared again");
+
+	/* The second goes away; the first keeps following. */
+	ioas_unmap_gmem(&w);
+	TEST_ASSERT(!__gmem_set_private(v.gmem, CONV_OFF, PAGE, &err),
+		    "conversion to private errno=%d", errno);
+	TEST_ASSERT(!vmm_iova_phys(&v, CONV_IOVA),
+		    "the first IOAS maps a private page");
+	TEST_ASSERT(!__gmem_set_shared(v.gmem, CONV_OFF, PAGE, &err),
+		    "conversion to shared errno=%d", errno);
+	TEST_ASSERT(vmm_iova_phys(&v, CONV_IOVA) == frame,
+		    "the first IOAS does not map the page shared again");
+
+	vmm_destroy(&v);
+}
+
+/*
+ * PFN mode: a read-only frame is mapped read only for the guest and the VMM,
+ * and stays mapped for the device.  The guest maps the page first, so that
+ * marking it read only must zap it.
+ */
+static void scenario_readonly(void)
+{
+	struct vmm v = { .name = "pfn" };
+	volatile uint64_t *page;
+
+	pr_info("   read-only frames on a PFN mode provider\n");
+	vmm_setup(&v, BACKING_PFN);
+	page = (volatile uint64_t *)(v.hva + RO_OFF);
+
+	*page = MAGIC;
+	vmm_run(&v, 0, 0, RO_GPA);
+	TEST_ASSERT(expect_sync_then_done(&v) == MAGIC,
+		    "guest does not read the host's write");
+
+	mps_set(v.child, MPS_SET_READONLY, RO_OFF, PAGE, true);
+	TEST_ASSERT(vmm_iova_mapped(&v, IOVA + RO_OFF, PAGE, true),
+		    "read-only page not mapped for the device");
+	vmm_run(&v, 0, 0, RO_GPA);
+	TEST_ASSERT(expect_sync_then_done(&v) == MAGIC,
+		    "guest cannot read a read-only page");
+	vmm_run(&v, RO_GPA, ~MAGIC, 0);
+	expect_memory_fault(&v, RO_GPA);
+	TEST_ASSERT(host_write_faults(page, ~MAGIC),
+		    "host write to a read-only page did not fault");
+	TEST_ASSERT(*page == MAGIC, "write to a read-only page landed");
+
+	/* Writable again: the guest retries the same write. */
+	mps_set(v.child, MPS_SET_READONLY, RO_OFF, PAGE, false);
+	vmm_resume(&v);
+	expect_done(&v);
+	TEST_ASSERT(*page == ~MAGIC, "write after clearing read only lost");
+	TEST_ASSERT(!host_write_faults(page, MAGIC),
+		    "host write after clearing read only faulted");
+
+	vmm_destroy(&v);
+}
+
+/* PFN mode: private frames, and the provider's changes. */
+static void scenario_pfn(void)
+{
+	struct vmm v = { .name = "pfn" };
+	volatile uint64_t *host;
+	uint64_t err = 0, frame;
+
+	pr_info("4. private pages on a PFN mode provider\n");
+	vmm_setup(&v, BACKING_PFN);
+	host = (volatile uint64_t *)(v.hva + PFN_OFF);
+
+	vmm_run(&v, PFN_GPA, MAGIC, 0);
+	expect_done(&v);
+
+	/* Private: the device and the host lose it, the guest keeps it. */
+	TEST_ASSERT(!__gmem_set_private(v.gmem, PFN_OFF, PAGE, &err),
+		    "conversion to private errno=%d", errno);
+	TEST_ASSERT(!vmm_iova_phys(&v, PFN_IOVA), "device maps a private page");
+	TEST_ASSERT(host_read_faults(host),
+		    "host mapping of a private page readable");
+	vmm_run(&v, 0, 0, PFN_GPA);
+	TEST_ASSERT(expect_sync_then_done(&v) == MAGIC,
+		    "the private page lost the guest's write");
+
+	/*
+	 * The provider takes the private page back: a hole for the guest until
+	 * it comes back, zeroed by the sample.
+	 */
+	mps_set(v.child, MPS_SET_PRESENT, PFN_OFF, PAGE, false);
+	vmm_run(&v, 0, 0, PFN_GPA);
+	expect_memory_fault(&v, PFN_GPA);
+	mps_set(v.child, MPS_SET_PRESENT, PFN_OFF, PAGE, true);
+	vmm_run(&v, 0, 0, PFN_GPA);
+	TEST_ASSERT(!expect_sync_then_done(&v),
+		    "the page came back with old contents");
+	TEST_ASSERT(!vmm_iova_phys(&v, PFN_IOVA),
+		    "device maps a private page given back");
+
+	/* A page that is not backed may become private; the guest faults. */
+	TEST_ASSERT(!__gmem_set_shared(v.gmem, PFN_OFF, PAGE, &err),
+		    "conversion to shared errno=%d", errno);
+	mps_set(v.child, MPS_SET_PRESENT, PFN_OFF, PAGE, false);
+	TEST_ASSERT(!__gmem_set_private(v.gmem, PFN_OFF, PAGE, &err),
+		    "conversion of an absent page errno=%d", errno);
+	vmm_run(&v, 0, 0, PFN_GPA);
+	expect_memory_fault(&v, PFN_GPA);
+	mps_set(v.child, MPS_SET_PRESENT, PFN_OFF, PAGE, true);
+	vmm_run(&v, PFN_GPA, MAGIC, 0);
+	expect_done(&v);
+
+	/* A read-only frame never backs a private page: a hole until writable. */
+	mps_set(v.child, MPS_SET_READONLY, PFN_OFF, PAGE, true);
+	vmm_run(&v, 0, 0, PFN_GPA);
+	expect_memory_fault(&v, PFN_GPA);
+	mps_set(v.child, MPS_SET_READONLY, PFN_OFF, PAGE, false);
+	vmm_run(&v, 0, 0, PFN_GPA);
+	TEST_ASSERT(expect_sync_then_done(&v) == MAGIC,
+		    "the private page lost its contents while read only");
+
+	/* Back to shared: the device maps it again, with the contents. */
+	TEST_ASSERT(!__gmem_set_shared(v.gmem, PFN_OFF, PAGE, &err),
+		    "conversion to shared errno=%d", errno);
+	TEST_ASSERT(vmm_iova_phys(&v, PFN_IOVA), "device does not map the page");
+	TEST_ASSERT(*host == MAGIC, "host does not see the page");
+
+	/* A shared page taken back and given back: the device follows. */
+	mps_set(v.child, MPS_SET_PRESENT, PFN_OFF, PAGE, false);
+	TEST_ASSERT(!vmm_iova_phys(&v, PFN_IOVA), "device maps an absent page");
+	TEST_ASSERT(host_read_faults(host), "host maps an absent page");
+	mps_set(v.child, MPS_SET_PRESENT, PFN_OFF, PAGE, true);
+	TEST_ASSERT(vmm_iova_phys(&v, PFN_IOVA),
+		    "device does not map a page given back");
+
+	/*
+	 * A punch hole gives the frame back to the provider, which zeroes it
+	 * and keeps backing the page: the device maps the same frame
+	 * afterwards, and the host reads it zeroed.
+	 */
+	frame = vmm_iova_phys(&v, PFN_IOVA);
+	*host = MAGIC;
+	TEST_ASSERT(!fallocate(v.gmem, FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE,
+			       PFN_OFF, PAGE),
+		    "punch hole errno=%d", errno);
+	TEST_ASSERT(vmm_iova_phys(&v, PFN_IOVA) == frame,
+		    "device does not map the page after a punch hole");
+	TEST_ASSERT(!*host, "a punched page kept its contents");
+
+	vmm_destroy(&v);
+}
+
+/*
+ * A private 2 MiB block of a PFN mode provider gets one stage-2 mapping, which a
+ * change inside the block takes away; the rest of the block stays readable.
+ */
+static void scenario_large(void)
+{
+	struct vmm v = { .name = "pfn" };
+	uint64_t before, after, err = 0;
+
+	pr_info("5. a 2 MiB private mapping on a PFN mode provider\n");
+	vmm_setup(&v, BACKING_PFN);
+
+	if (vmm_iova_phys(&v, IOVA) & (HPAGE - 1)) {
+		pr_info("   skipped: the file's frames are not 2 MiB aligned\n");
+		vmm_destroy(&v);
+		return;
+	}
+
+	/* Fault in the guest's code and stack first, then count. */
+	vmm_run(&v, 0, 0, 0);
+	expect_done(&v);
+	TEST_ASSERT(!__gmem_set_private(v.gmem, LARGE_OFF, HPAGE, &err),
+		    "conversion of a 2 MiB block errno=%d", errno);
+	TEST_ASSERT(vmm_iova_mapped(&v, LARGE_IOVA, HPAGE, false),
+		    "device maps part of a private block");
+	before = vm_get_stat(v.vm, pages_2m);
+
+	vmm_run(&v, LARGE_GPA, MAGIC, 0);
+	expect_done(&v);
+	after = vm_get_stat(v.vm, pages_2m);
+	TEST_ASSERT(after == before + 1, "%llu 2 MiB mappings, want %llu",
+		    (unsigned long long)after, (unsigned long long)before + 1);
+
+	/*
+	 * A page taken out of the block removes the 2 MiB mapping; the other
+	 * pages still work.
+	 */
+	mps_set(v.child, MPS_SET_PRESENT, LARGE_OFF + PAGE, PAGE, false);
+	TEST_ASSERT(vm_get_stat(v.vm, pages_2m) == before,
+		    "the 2 MiB mapping survived a change inside it");
+	vmm_run(&v, 0, 0, LARGE_GPA);
+	TEST_ASSERT(expect_sync_then_done(&v) == MAGIC,
+		    "the rest of the block lost the guest's write");
+	TEST_ASSERT(vm_get_stat(v.vm, pages_2m) == before,
+		    "a 2 MiB mapping over a block with a hole");
+	vmm_run(&v, 0, 0, LARGE_GPA + PAGE);
+	expect_memory_fault(&v, LARGE_GPA + PAGE);
+	mps_set(v.child, MPS_SET_PRESENT, LARGE_OFF + PAGE, PAGE, true);
+	vmm_run(&v, 0, 0, LARGE_GPA + PAGE);
+	TEST_ASSERT(!expect_sync_then_done(&v),
+		    "a page given back kept its contents");
+
+	/* Shared again: the device maps the whole block. */
+	TEST_ASSERT(!__gmem_set_shared(v.gmem, LARGE_OFF, HPAGE, &err),
+		    "conversion to shared errno=%d", errno);
+	TEST_ASSERT(vmm_iova_mapped(&v, LARGE_IOVA, HPAGE, true),
+		    "device does not map the shared block");
+
+	vmm_destroy(&v);
+}
+
+/*
+ * A provider that backs a hole with the whole 2 MiB block around it: the
+ * device maps the block, whether a host fault or fallocate() asks for one
+ * page of it.
+ */
+static void scenario_fill(void)
+{
+	struct mps_new_file nf = { .flags = MPS_NEW_FILE_FILL,
+				   .size = FILE_SIZE };
+	struct vmm v = { .name = "pfn" };
+	volatile uint64_t *host;
+	int fd;
+
+	pr_info("6. a provider that backs 2 MiB blocks on demand\n");
+	fd = ioctl(ctl, MPS_NEW_FILE, &nf);
+	TEST_ASSERT(fd >= 0, "MPS_NEW_FILE(FILL) errno=%d", errno);
+	vmm_create_type(&v, KVM_X86_SW_PROTECTED_VM);
+	v.pfn = true;
+	vmm_attach(&v, fd);
+
+	mps_set(v.child, MPS_SET_PRESENT, 0, FILE_SIZE, false);
+	TEST_ASSERT(vmm_iova_mapped(&v, IOVA, FILE_SIZE, false),
+		    "device maps absent pages");
+
+	host = (volatile uint64_t *)(v.hva + LARGE_OFF + 5 * PAGE);
+	TEST_ASSERT(!*host, "a page backed on demand is not zeroed");
+	TEST_ASSERT(vmm_iova_mapped(&v, LARGE_IOVA, HPAGE, true),
+		    "device does not map the block backed for a host fault");
+
+	TEST_ASSERT(!fallocate(v.gmem, FALLOC_FL_KEEP_SIZE, 3 * PAGE, PAGE),
+		    "fallocate errno=%d", errno);
+	TEST_ASSERT(vmm_iova_mapped(&v, IOVA, HPAGE, true),
+		    "device does not map the block backed for fallocate()");
+
+	/*
+	 * A punch hole gives the block back and the provider frees it: the
+	 * device sees a hole until the next access backs the block again.
+	 */
+	*host = MAGIC;
+	TEST_ASSERT(!fallocate(v.gmem, FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE,
+			       LARGE_OFF, HPAGE),
+		    "punch hole errno=%d", errno);
+	TEST_ASSERT(vmm_iova_mapped(&v, LARGE_IOVA, HPAGE, false),
+		    "device maps a block the provider freed");
+	TEST_ASSERT(vmm_iova_mapped(&v, IOVA, HPAGE, true),
+		    "device lost the block next to the punch hole");
+	TEST_ASSERT(!*host, "a block backed again is not zeroed");
+	TEST_ASSERT(vmm_iova_mapped(&v, LARGE_IOVA, HPAGE, true),
+		    "device does not map the block backed again");
+
+	vmm_destroy(&v);
+}
+
+int main(void)
+{
+	TEST_REQUIRE(kvm_check_cap(KVM_CAP_VM_TYPES) &
+		     BIT(KVM_X86_SW_PROTECTED_VM));
+	TEST_REQUIRE(kvm_check_cap(KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES));
+	TEST_REQUIRE(kvm_check_cap(KVM_CAP_GUEST_MEMFD_FLAGS) &
+		     GUEST_MEMFD_FLAG_USE_RESOURCE);
+
+	scenario_convert(BACKING_FOLIO);
+	scenario_alloc(BACKING_FOLIO);
+	scenario_convert(BACKING_TMPFS);
+	scenario_alloc(BACKING_TMPFS);
+	ioas_check_map_overflow();
+	scenario_poison(BACKING_FOLIO);
+	scenario_poison(BACKING_TMPFS);
+	scenario_two_ioas();
+
+	ctl = open("/dev/mem_provider_sample", O_RDWR);
+	if (ctl < 0) {
+		pr_info("PFN mode scenarios skipped: mem_provider_sample not loaded\n");
+		return 0;
+	}
+
+	scenario_convert(BACKING_PFN);
+	scenario_readonly();
+	scenario_pfn();
+	scenario_large();
+	scenario_fill();
+
+	close(ctl);
+	pr_info("gmem_conversion_iommufd: all scenarios passed\n");
+	return 0;
+}
-- 
2.47.3

Re: [RFC PATCH 11/14] KVM: guest_memfd: Add kvm_gmem_restore() for PFN mode providers

[Ackerley Tng]

Fred Griffoul <griffoul@gmail.com> writes:

> From: Fred Griffoul <fgriffo@amazon.co.uk>
>
> A device cannot fault, so when a PFN mode provider gives frames to pages
> that had none, or makes a read-only frame writable, a subsystem that
> maps the file must be told.

Totally makes sense that guest_memfd needs a way to actively map pages
into the IOMMU page tables, rather than passively like for userspace
page tables and stage 2 page tables.

When the memory is owned by guest_memfd, on a to-shared conversion,
guest_memfd would do the mapping for IOMMU, which can't handle faults
like Stage 2 or userspace page tables.

> kvm_gmem_revoke() does not fit: it would
> zap the guest's and the VMM's mappings and reclaim frames that a guest
> may already use.
>

I'd drop this paragraph from the commit message, kvm_gmem_revoke() is
the complete opposite and isn't a candidate at all for the goal, which
is to notify importers that a page is now available (and they can map it
in).

> Add kvm_gmem_restore(), the counterpart to kvm_gmem_revoke() that the
> sample provider leaves for users that cannot fault. It only updates the
> exports, under the shared invalidate lock.
>
> Assisted-by: Claude:claude-opus-5-5
> Signed-off-by: Fred Griffoul <fgriffo@amazon.co.uk>
> ---
>  arch/x86/kvm/Makefile       |  1 +
>  include/linux/guest_memfd.h | 37 ++++++++++++++++++++++---------------
>  virt/kvm/guest_memfd.c      | 35 +++++++++++++++++++++++++++++++++--
>  3 files changed, 56 insertions(+), 17 deletions(-)
>
> diff --git a/arch/x86/kvm/Makefile b/arch/x86/kvm/Makefile
> index 25fb3d6e2032..3255280884b5 100644
> --- a/arch/x86/kvm/Makefile
> +++ b/arch/x86/kvm/Makefile
> @@ -63,6 +63,7 @@ exports_grep_trailer := --include='*.[ch]' -nrw $(srctree)/virt/kvm $(srctree)/a
>  				-e kvm_write_track_remove_gfn \
>  				-e kvm_file_to_kvm_fn \
>  				-e kvm_gmem_revoke \
> +				-e kvm_gmem_restore \

provider --1--> guest_memfd --2--> IOMMU

2 is definitely required, but why is 1 required?

Between the first .get_pfn() or .lookup_pfn_range() () and
.put_pfn_range() or kvm_gmem_revoke(), the provider should consider the
memory in-use by guest_memfd.

When it's in-use by guest_memfd, then guest_memfd should use --2--> to
map into IOMMU page tables as necessary.

When it's not in-use by guest_memfd, why would the provider force it
into guest_memfd?

If the guest VM knows that the page is now usable (some other
communication channel), it would touch the memory and do a .get_pfn(),
that would be successful this time. Perhaps that's when --2--> needs to
be performed?

>  				-e kvm_gmem_export
>
>  # Force grep to emit a goofy group separator that can in turn be replaced with
>
> [...snip...]
>

Re: [RFC PATCH 13/14] samples/kvm: Back absent pages on demand in the sample provider

[Ackerley Tng]

Fred Griffoul <griffoul@gmail.com> writes:

> From: Fred Griffoul <fgriffo@amazon.co.uk>
>
> The sample provider fails get_pfn() for an absent page and keeps its
> frames on a punch hole, so tests cannot exercise a provider that backs
> a hole when it is asked for a frame, one that backs more than the page
> asked for, or one that frees the frames that a punch hole gives back.
>
> Add MPS_NEW_FILE_FILL: get_pfn() then backs an absent page with the
> whole 2 MiB block around it, and put_pfn_range() makes the pages absent
> again.
>

Can what MPS_NEW_FILE_FILL is meant to illustrate be illustrated with
the ioctl that sets memory present?

> Assisted-by: Claude:claude-opus-5-5
> Signed-off-by: Fred Griffoul <fgriffo@amazon.co.uk>
>
> [...snip...]
>

Re: [RFC PATCH 01/14] KVM: guest_memfd: Clamp the order of PFN mode blocks to what KVM can map

[Ackerley Tng]

Fred Griffoul <griffoul@gmail.com> writes:

> From: Fred Griffoul <fgriffo@amazon.co.uk>
>
> KVM maps guest memory in 4K, 2M or 1G blocks, and uses guest_memfd's
> max_order to choose. A folio of order N is physically aligned and never
> mixes shared and private pages, so the order needs no check. A PFN mode
> provider's frames need not be aligned

I'll need to think about max_order > 0 more for PFNs. I put it out of my
mind because my frame of mind is that only order-0 folios are supported
today. Perhaps with PFNs it's actually a lot easier to support order >
0.

> and the provider does not know
> which pages are private, yet kvm_gmem_get_pfn() passes its order to KVM
> unchanged. KVM can then map a block over the wrong frames, or with one
> attribute over shared and private pages.
>

Yes, makes sense that private/shared contiguity needs to be taken into
account. I didn't consider this because order-0 is always safe to
map. Thank you!

>
> [...snip...]
>
> -static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start,
> +static void __kvm_gmem_invalidate_start(struct inode *inode,
> +					struct gmem_file *f, pgoff_t start,
>  					pgoff_t end,
>  					enum kvm_gfn_range_filter attr_filter)
>  {
> +	const gfn_t blk = kvm_gmem_is_pfn_mode(GMEM_I(inode)) ?
> +			  1UL << PUD_ORDER : 1;
>  	bool flush = false, found_memslot = false;
>  	struct kvm_memory_slot *slot;
>  	struct kvm *kvm = f->kvm;
> @@ -355,6 +360,16 @@ static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start,
>
>  		flush |= kvm_mmu_unmap_gfn_range(kvm, &gfn_range);
>
> +		/*
> +		 * In PFN mode, a fault may map a block of up to PUD_ORDER
> +		 * pages, sized before this invalidation (see
> +		 * kvm_gmem_pfn_map_order()): make faults anywhere in the
> +		 * blocks around the range retry.
> +		 */
> +		kvm_mmu_invalidate_range_add(kvm,
> +					     round_down(gfn_range.start, blk),
> +					     round_up(gfn_range.end, blk));
> +

Yes, we want to make the invalidation range larger, but not the range
being zapped. We can't over-zap for CoCo VMs.

Would it be better to refactor kvm_mmu_unmap_gfn_range() instead of
doing kvm_mmu_invalidate_range_add() twice?

>  #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
>  		kvm_arch_gmem_invalidate_range(kvm, &gfn_range);
>  #endif
> @@ -374,7 +389,7 @@ static void kvm_gmem_invalidate_start(struct inode *inode, pgoff_t start,
>  	struct gmem_file *f;
>
>  	kvm_gmem_for_each_file(f, inode)
> -		__kvm_gmem_invalidate_start(f, start, end, filter);
> +		__kvm_gmem_invalidate_start(inode, f, start, end, filter);
>  }
>
>  static void __kvm_gmem_invalidate_end(struct gmem_file *f, pgoff_t start,
> @@ -670,7 +685,7 @@ static int kvm_gmem_release(struct inode *inode, struct file *file)
>  	 * Zap all SPTEs pointed at by this file.  Do not free the backing
>  	 * memory, as its lifetime is associated with the inode, not the file.
>  	 */
> -	__kvm_gmem_invalidate_start(f, 0, -1ul,
> +	__kvm_gmem_invalidate_start(inode, f, 0, -1ul,
>  				    kvm_gmem_get_all_gfns_filter(inode));
>  	__kvm_gmem_invalidate_end(f, 0, -1ul);
>
> @@ -1535,6 +1550,44 @@ static struct folio *__kvm_gmem_get_pfn(struct file *file,
>  	return folio;
>  }
>
> +/*
> + * The order a PFN mode block can be mapped at: the provider's order, no
> + * larger than the alignment of @gfn and @pfn to @index allows, rounded down to
> + * a page-table size, and never over pages that are not all shared or all
> + * private, since KVM maps a block with a single attribute.  Called with the
> + * invalidate lock held; an invalidation of any page of the block makes the
> + * fault retry (see __kvm_gmem_invalidate_start()).
> + */
> +static int kvm_gmem_pfn_map_order(struct inode *inode, pgoff_t index,
> +				  gfn_t gfn, kvm_pfn_t pfn, int order)
> +{
> +	u64 misalign = (gfn ^ index) | (pfn ^ index);
> +	u64 attrs;
> +
> +	if (WARN_ON_ONCE(order < 0))
> +		return 0;
> +
> +	order = min(order, PUD_ORDER);
> +	if (misalign)
> +		order = min_t(int, order, __ffs64(misalign));
> +
> +	if (order >= PUD_ORDER)
> +		order = PUD_ORDER;
> +	else if (order >= PMD_ORDER)
> +		order = PMD_ORDER;
> +	else
> +		return 0;
> +
> +	attrs = kvm_gmem_get_attributes(inode,
> +					mtree_load(&GMEM_I(inode)->attributes, index));
> +	while (order &&
> +	       !kvm_gmem_range_has_attributes(inode,
> +					      ALIGN_DOWN(index, 1UL << order),
> +					      1UL << order, attrs))
> +		order = order == PUD_ORDER ? PMD_ORDER : 0;
> +	return order;
> +}
> +

I'll review this function more closely again. I think the mapping order
should fit into how KVM does it. After taking guest_memfd's input on
max_order, KVM then lets these parties possibly tune the max order down:

1. lpage_info contains information about highest order due to memslot
   alignment and alignment due to VM-level memory attributes
2. CoCo platform can impose a max order
3. Userspace mappings can impose a max order too.

So I think we should follow the same model:

+ The provider reports the max_order based on it's contiguity
  information, which is the actual contiguity of memory and
  read-only attributes
+ guest_memfd further reduces max_order due to shared/private contiguity
+ After handing it to KVM, KVM further reduces the max order based on
  the above.

I think mappings for all other page tables (userspace and IOMMU) should
also follow this approach.

I guess the kvm_gmem_fault_user_mapping() equivalent of this max_order
tuning is missing from this series. :) I think we should defer order > 0
to a later series though. I think read-only-ness should be part of the
first PFN mode series.

>
> [...snip...]
>

Re: [RFC PATCH 11/14] KVM: guest_memfd: Add kvm_gmem_restore() for PFN mode providers

[Frederic Griffoul]

On Thu, Oct 29, 2026 at 05:52:40AM -0700, Ackerley Tng wrote:
> Fred Griffoul <griffoul@gmail.com> writes:
>
> > kvm_gmem_revoke() does not fit: it would
> > zap the guest's and the VMM's mappings and reclaim frames that a guest
> > may already use.
> >
>
> I'd drop this paragraph from the commit message, kvm_gmem_revoke() is
> the complete opposite and isn't a candidate at all for the goal, which
> is to notify importers that a page is now available (and they can map it
> in).

Sure, I'll drop it.

> provider --1--> guest_memfd --2--> IOMMU
>
> 2 is definitely required, but why is 1 required?
>
> Between the first .get_pfn() or .lookup_pfn_range() () and
> .put_pfn_range() or kvm_gmem_revoke(), the provider should consider the
> memory in-use by guest_memfd.
>
> When it's in-use by guest_memfd, then guest_memfd should use --2--> to
> map into IOMMU page tables as necessary.
>
> When it's not in-use by guest_memfd, why would the provider force it
> into guest_memfd?
>
> If the guest VM knows that the page is now usable (some other
> communication channel), it would touch the memory and do a .get_pfn(),
> that would be successful this time. Perhaps that's when --2--> needs to
> be performed?

The IOMMU never asks, and the CPU may not touch the page before a
device does. A few cases where only the provider knows:

- The provider gives frames back with no guest involved: a page that
  arrives during post-copy, a range moved back to the VM, memory given
  to an overcommitted VM.

- The guest already gave that IOVA to a device as a DMA buffer. The
  device writes it without any CPU access, so there is no .get_pfn()
  from a stage-2 fault, and most devices cannot fault.

- A read-only frame becomes writable. KVM finds out on the next write
  fault, but a device write to a read-only IOPTE just fails.

kvm_gmem_restore() doesn't push the memory into guest_memfd: guest_memfd
only passes it on to its exports. Stage-2 and userspace mappings stay
lazy, and iommufd looks the range up again and maps it.

I'll make that clear in the kernel-doc and the commit message. Would
that work for you?

Fred

Re: [RFC PATCH 01/14] KVM: guest_memfd: Clamp the order of PFN mode blocks to what KVM can map

[Frederic Griffoul]

On Thu, Oct 29, 2026 at 06:25:49AM -0700, Ackerley Tng wrote:
> Fred Griffoul <griffoul@gmail.com> writes:
>
> > +        kvm_mmu_invalidate_range_add(kvm,
> > +                         round_down(gfn_range.start, blk),
> > +                         round_up(gfn_range.end, blk));
> > +
>
> Yes, we want to make the invalidation range larger, but not the range
> being zapped. We can't over-zap for CoCo VMs.
>
> Would it be better to refactor kvm_mmu_unmap_gfn_range() instead of
> doing kvm_mmu_invalidate_range_add() twice?

Good point. As far as I can see, kvm_mmu_unmap_gfn_range() is only
kvm_mmu_invalidate_range_add() plus kvm_unmap_gfn_range(), so
guest_memfd could do the two steps itself: add the blocks around the
range, then unmap the exact range. The zap stays exact and the range is
added once. Or am I missing something about kvm_mmu_unmap_gfn_range()
that a refactor would need to keep?

> So I think we should follow the same model:
>
> + The provider reports the max_order based on it's contiguity
>   information, which is the actual contiguity of memory and
>   read-only attributes
> + guest_memfd further reduces max_order due to shared/private contiguity
> + After handing it to KVM, KVM further reduces the max order based on
>   the above.
>
> I think mappings for all other page tables (userspace and IOMMU) should
> also follow this approach.

Agreed. guest_memfd should then only reduce the order for shared and
private pages, and drop the gfn alignment and the rounding to a
page-table size, which KVM does. For the provider, "contiguity" would
mean a naturally aligned block of contiguous frames with the same
attributes, the frames aligned to the block too, so a mapping of that
size is valid as is.

One gap: KVM does not cover a memslot whose gfn and guest_memfd offset
are not aligned to each other. On x86, kvm_alloc_memslot_metadata()
disallows large pages when base_gfn and userspace_addr are misaligned,
but nothing compares base_gfn with gmem.pgoff, and kvm_gmem_bind()
takes any page-aligned offset. A gmem.pgoff check next to the
userspace_addr one would keep that in KVM too.

The export lookup in the series already works this way: it reports the
provider's block, reduced for shared and private pages. iommufd maps
PAGE_SIZE IOPTEs in this series, and will pick larger IOPTEs from that
block later.

> I guess the kvm_gmem_fault_user_mapping() equivalent of this max_order
> tuning is missing from this series. :) I think we should defer order >
> 0 to a later series though. I think read-only-ness should be part of
> the first PFN mode series.

Right, userspace mappings are PAGE_SIZE for now.

Deferring order > 0 for KVM and userspace is fine with me. I'd like to
keep the order in the export lookup, though: large IOPTEs matter for
device performance, and I'm preparing iommupt changes to split and merge
them in place.

Fred

On Thu, Oct 29, 2026 at 1:25 PM Ackerley Tng <ackerleytng@google.com> wrote:
>
> Fred Griffoul <griffoul@gmail.com> writes:
>
> > From: Fred Griffoul <fgriffo@amazon.co.uk>
> >
> > KVM maps guest memory in 4K, 2M or 1G blocks, and uses guest_memfd's
> > max_order to choose. A folio of order N is physically aligned and never
> > mixes shared and private pages, so the order needs no check. A PFN mode
> > provider's frames need not be aligned
>
> I'll need to think about max_order > 0 more for PFNs. I put it out of my
> mind because my frame of mind is that only order-0 folios are supported
> today. Perhaps with PFNs it's actually a lot easier to support order >
> 0.
>
> > and the provider does not know
> > which pages are private, yet kvm_gmem_get_pfn() passes its order to KVM
> > unchanged. KVM can then map a block over the wrong frames, or with one
> > attribute over shared and private pages.
> >
>
> Yes, makes sense that private/shared contiguity needs to be taken into
> account. I didn't consider this because order-0 is always safe to
> map. Thank you!
>
> >
> > [...snip...]
> >
> > -static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start,
> > +static void __kvm_gmem_invalidate_start(struct inode *inode,
> > +                                     struct gmem_file *f, pgoff_t start,
> >                                       pgoff_t end,
> >                                       enum kvm_gfn_range_filter attr_filter)
> >  {
> > +     const gfn_t blk = kvm_gmem_is_pfn_mode(GMEM_I(inode)) ?
> > +                       1UL << PUD_ORDER : 1;
> >       bool flush = false, found_memslot = false;
> >       struct kvm_memory_slot *slot;
> >       struct kvm *kvm = f->kvm;
> > @@ -355,6 +360,16 @@ static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start,
> >
> >               flush |= kvm_mmu_unmap_gfn_range(kvm, &gfn_range);
> >
> > +             /*
> > +              * In PFN mode, a fault may map a block of up to PUD_ORDER
> > +              * pages, sized before this invalidation (see
> > +              * kvm_gmem_pfn_map_order()): make faults anywhere in the
> > +              * blocks around the range retry.
> > +              */
> > +             kvm_mmu_invalidate_range_add(kvm,
> > +                                          round_down(gfn_range.start, blk),
> > +                                          round_up(gfn_range.end, blk));
> > +
>
> Yes, we want to make the invalidation range larger, but not the range
> being zapped. We can't over-zap for CoCo VMs.
>
> Would it be better to refactor kvm_mmu_unmap_gfn_range() instead of
> doing kvm_mmu_invalidate_range_add() twice?
>
> >  #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
> >               kvm_arch_gmem_invalidate_range(kvm, &gfn_range);
> >  #endif
> > @@ -374,7 +389,7 @@ static void kvm_gmem_invalidate_start(struct inode *inode, pgoff_t start,
> >       struct gmem_file *f;
> >
> >       kvm_gmem_for_each_file(f, inode)
> > -             __kvm_gmem_invalidate_start(f, start, end, filter);
> > +             __kvm_gmem_invalidate_start(inode, f, start, end, filter);
> >  }
> >
> >  static void __kvm_gmem_invalidate_end(struct gmem_file *f, pgoff_t start,
> > @@ -670,7 +685,7 @@ static int kvm_gmem_release(struct inode *inode, struct file *file)
> >        * Zap all SPTEs pointed at by this file.  Do not free the backing
> >        * memory, as its lifetime is associated with the inode, not the file.
> >        */
> > -     __kvm_gmem_invalidate_start(f, 0, -1ul,
> > +     __kvm_gmem_invalidate_start(inode, f, 0, -1ul,
> >                                   kvm_gmem_get_all_gfns_filter(inode));
> >       __kvm_gmem_invalidate_end(f, 0, -1ul);
> >
> > @@ -1535,6 +1550,44 @@ static struct folio *__kvm_gmem_get_pfn(struct file *file,
> >       return folio;
> >  }
> >
> > +/*
> > + * The order a PFN mode block can be mapped at: the provider's order, no
> > + * larger than the alignment of @gfn and @pfn to @index allows, rounded down to
> > + * a page-table size, and never over pages that are not all shared or all
> > + * private, since KVM maps a block with a single attribute.  Called with the
> > + * invalidate lock held; an invalidation of any page of the block makes the
> > + * fault retry (see __kvm_gmem_invalidate_start()).
> > + */
> > +static int kvm_gmem_pfn_map_order(struct inode *inode, pgoff_t index,
> > +                               gfn_t gfn, kvm_pfn_t pfn, int order)
> > +{
> > +     u64 misalign = (gfn ^ index) | (pfn ^ index);
> > +     u64 attrs;
> > +
> > +     if (WARN_ON_ONCE(order < 0))
> > +             return 0;
> > +
> > +     order = min(order, PUD_ORDER);
> > +     if (misalign)
> > +             order = min_t(int, order, __ffs64(misalign));
> > +
> > +     if (order >= PUD_ORDER)
> > +             order = PUD_ORDER;
> > +     else if (order >= PMD_ORDER)
> > +             order = PMD_ORDER;
> > +     else
> > +             return 0;
> > +
> > +     attrs = kvm_gmem_get_attributes(inode,
> > +                                     mtree_load(&GMEM_I(inode)->attributes, index));
> > +     while (order &&
> > +            !kvm_gmem_range_has_attributes(inode,
> > +                                           ALIGN_DOWN(index, 1UL << order),
> > +                                           1UL << order, attrs))
> > +             order = order == PUD_ORDER ? PMD_ORDER : 0;
> > +     return order;
> > +}
> > +
>
> I'll review this function more closely again. I think the mapping order
> should fit into how KVM does it. After taking guest_memfd's input on
> max_order, KVM then lets these parties possibly tune the max order down:
>
> 1. lpage_info contains information about highest order due to memslot
>    alignment and alignment due to VM-level memory attributes
> 2. CoCo platform can impose a max order
> 3. Userspace mappings can impose a max order too.
>
> So I think we should follow the same model:
>
> + The provider reports the max_order based on it's contiguity
>   information, which is the actual contiguity of memory and
>   read-only attributes
> + guest_memfd further reduces max_order due to shared/private contiguity
> + After handing it to KVM, KVM further reduces the max order based on
>   the above.
>
> I think mappings for all other page tables (userspace and IOMMU) should
> also follow this approach.
>
> I guess the kvm_gmem_fault_user_mapping() equivalent of this max_order
> tuning is missing from this series. :) I think we should defer order > 0
> to a later series though. I think read-only-ness should be part of the
> first PFN mode series.
>
> >
> > [...snip...]
> >