[RFC PATCH v3 00/13] KVM: guest_memfd: Add a common framework for device memory providers

[RFC PATCH v3 00/13] KVM: guest_memfd: Add a common framework for device memory providers

[Aneesh Kumar K.V (Arm)]

Confidential device assignment needs guest_memfd to manage device MMIO as
well as RAM. Device resources such as PCI BARs supply PFNs directly, and
their protected mappings need architecture-specific map and unmap
operations.

This series adds the common guest_memfd support needed by device memory
providers. It builds on the resource-provider API [1] and the existing
folio and RAM PFN backing support.

The VFIO provider, IOMMUFD integration and Arm CCA implementation are
available together at:

  https://git.gitlab.arm.com/linux-arm/linux-cca.git cca/topics/cca-da-gmem-backend

This series depends on the resource-provider groundwork from:
[1] https://lore.kernel.org/all/20261022-gmem-alternative-providers-v2-0-5dd1dc6c660a@google.com

Notes:
* Codex assisted with implementation, review and series organization.

Changes from v2:
https://lore.kernel.org/all/20261017073149.1303738-1-aneesh.kumar@kernel.org
* Rebase on the v2 alternative-provider series and separate the common
  guest_memfd framework from the VFIO, IOMMUFD and Arm CCA integration.
* Introduce backing operations to separate folio, RAM PFN and device MMIO
  handling while sharing locking, invalidation and attribute updates.
* Replace prepare_conversion()/finish_conversion() with non-failing
  lock()/unlock() callbacks.
* Convert device memory through ordinary file-offset attribute updates.
  Derive GPA from memslot bindings and PA from provider PFNs, removing the
  device request object.
* Allow shared and private device PFN lookup in the common layer. Include
  both mapping types in generic invalidation and leave architecture
  mapping policy to the backend.
* Support device range revocation through the common revoke path without
  reclaiming MMIO PFNs or changing attributes.
* Add shared device mmap through the existing get_pfn() fault path. Keep
  VMA protections and flags in the core
* Require only get_pfn() for device providers; lookup_pfn_range() remains
  required for RAM PFN providers.

Changes from v1:
https://lore.kernel.org/all/20261010072504.536230-1-aneesh.kumar@kernel.org
* Require GUEST_MEMFD_FLAG_MMAP to allow VFIO BAR mmap while a provider is
  attached; disable it by default.
* Use the common guest_memfd_provider_operations for folio and PFN
  providers. Select operations through the resource mapping superblock
  and forward VFIO operations to the device driver.

Cc: Ackerley Tng <ackerleytng@google.com>
Cc: Alex Williamson <alex@shazbot.org>
Cc: David Woodhouse <dwmw2@infradead.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: "Joerg Roedel (AMD)" <joro@8bytes.org>
Cc: Kevin Tian <kevin.tian@intel.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Robin Murphy <robin.murphy@arm.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Will Deacon <will@kernel.org>
Cc: Alexey Kardashevskiy <aik@amd.com>
Cc: Xu Yilun <yilun.xu@linux.intel.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
Cc: Steven Price <steven.price@arm.com>
Cc: Fred Griffoul <griffoul@gmail.com>
Cc: iommu@lists.linux.dev
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev


Aneesh Kumar K.V (Arm) (13):
  KVM: guest_memfd: Let providers prepare memory regions
  KVM: guest_memfd: Dispatch attribute conversion through backing
    operations
  KVM: guest_memfd: Let providers coordinate mapping changes
  KVM: guest_memfd: Dispatch revocation through backing operations
  KVM: guest_memfd: Split population by backing type
  KVM: guest_memfd: Support device-backed PFN providers
  KVM: guest_memfd: Support private/shared conversion of device memory
  KVM: guest_memfd: Track private device ranges after attribute updates
  KVM: guest_memfd: Expose device provider private state
  KVM: guest_memfd: Support range revocation of device mappings
  KVM: guest_memfd: Add shared device mmap through PFN providers
  KVM: guest_memfd: Remove device mappings on final close
  KVM: guest_memfd: Invalidate guest_memfd mappings before removing
    memslot bindings

 Documentation/virt/kvm/api.rst |  21 +-
 include/linux/guest_memfd.h    |  84 +++-
 include/linux/kvm_host.h       |   7 +
 virt/kvm/guest_memfd.c         | 690 ++++++++++++++++++++++++++-------
 4 files changed, 660 insertions(+), 142 deletions(-)

-- 
2.43.0

[RFC PATCH v3 01/13] KVM: guest_memfd: Let providers prepare memory regions

[Aneesh Kumar K.V (Arm)]

Memory providers may need to check memslot flags and backing ranges
before KVM installs a memslot. Add an optional prepare_memory_region()
callback to the common provider operations

Pass the byte offset, size and requested dirty-logging and read-only
flags, and return any error before publishing the slot. Providers do not
need access to KVM's memslot structure or internal flags. Existing
providers that do not implement the callback keep their current
behavior.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 include/linux/guest_memfd.h |  8 ++++++++
 virt/kvm/guest_memfd.c      | 17 ++++++++++++++++-
 2 files changed, 24 insertions(+), 1 deletion(-)

diff --git a/include/linux/guest_memfd.h b/include/linux/guest_memfd.h
index 3e2c0714ca33..17407504307b 100644
--- a/include/linux/guest_memfd.h
+++ b/include/linux/guest_memfd.h
@@ -14,6 +14,12 @@ struct mempolicy;
  * @attach: Attach to a resource file, perform filesystem-specific validation,
  *          and return an opaque provider context.
  * @release: Release provider context and unpin any resources.
+ * @prepare_memory_region: Optional check of a memory range and preparation of
+ *                         access to its backing memory. Called after common
+ *                         file checks and before publishing the memslot.
+ *                         Offset and size are in bytes. The flags contain only
+ *                         KVM_MEM_LOG_DIRTY_PAGES and KVM_MEM_READONLY bits.
+ *                         Return 0 on success or a negative error code.
  * @alloc_folio: Allocate an uninserted folio for a given index and NUMA policy.
  *               Must return a pointer to a struct folio on success, or an
  *               ERR_PTR() error code on failure. Must never return NULL.
@@ -44,6 +50,8 @@ struct mempolicy;
 struct guest_memfd_provider_operations {
 	void *(*attach)(struct inode *gmem_inode, struct file *resource_file);
 	void (*release)(void *provider);
+	int (*prepare_memory_region)(void *provider, u64 offset, u64 size,
+				     u32 flags);
 	struct folio *(*alloc_folio)(void *provider, pgoff_t index,
 				     struct mempolicy *mpol);
 	void (*invalidate_folio)(void *provider, struct folio *folio);
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index db568ffaebd4..0aae8f9cef88 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -1374,10 +1374,11 @@ int kvm_gmem_prepare_memory_region(struct kvm *kvm, struct kvm_memory_slot *slot
 				   unsigned int fd, uoff_t offset)
 {
 	uoff_t size = slot->npages << PAGE_SHIFT;
+	struct gmem_inode *gi;
 	struct gmem_file *f;
 	struct inode *inode;
 	struct file *file;
-
+	int ret;
 
 	BUILD_BUG_ON(sizeof(gpa_t) != sizeof(offset));
 	BUILD_BUG_ON(sizeof(gfn_t) != sizeof(slot->gmem.pgoff));
@@ -1397,10 +1398,24 @@ int kvm_gmem_prepare_memory_region(struct kvm *kvm, struct kvm_memory_slot *slot
 		goto err;
 
 	inode = file_inode(file);
+	gi = GMEM_I(inode);
 
 	if (!PAGE_ALIGNED(offset) || offset + size > i_size_read(inode))
 		goto err;
 
+	if (gi->provider_ops->prepare_memory_region) {
+		u32 flags = slot->flags & (KVM_MEM_LOG_DIRTY_PAGES | KVM_MEM_READONLY);
+
+		filemap_invalidate_lock_shared(inode->i_mapping);
+		ret = gi->provider_ops->prepare_memory_region(gi->provider, offset,
+							      size, flags);
+		filemap_invalidate_unlock_shared(inode->i_mapping);
+		if (ret) {
+			fput(file);
+			return ret;
+		}
+	}
+
 	/*
 	 * memslots of flag KVM_MEM_GUEST_MEMFD are immutable to change, so
 	 * kvm_gmem_bind() must occur on a new memslot.  Because the memslot
-- 
2.43.0

[RFC PATCH v3 02/13] KVM: guest_memfd: Dispatch attribute conversion through backing operations

[Aneesh Kumar K.V (Arm)]

Device backing needs different validation and architecture conversion
from RAM, while sharing the same locking, guest invalidation and
attribute publication sequence.

Introduce backing operations for prepare_attributes(), make_private()
and make_shared(), selected after provider attachment. Move the existing
folio and RAM PFN handling behind these operations.

Route fault-time private RAM preparation through prepare_private_pfn()
as well. Folio and RAM PFN backing retain the existing architecture
conversion hook; device backing can omit it without a device check in
the common PFN lookup path.

This provides the common sequence and backing-specific hooks needed by
the subsequent device-memory support.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 virt/kvm/guest_memfd.c | 225 ++++++++++++++++++++++++++++++-----------
 1 file changed, 164 insertions(+), 61 deletions(-)

diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 0aae8f9cef88..bd8324b4baba 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -33,6 +33,16 @@ struct gmem_file {
 
 static inline struct file *kvm_gmem_get_file(struct kvm_memory_slot *slot);
 
+struct kvm_gmem_backing_ops {
+	int (*prepare_attributes)(struct inode *inode, pgoff_t start,
+			size_t nr_pages, u64 attrs, pgoff_t *err_index);
+	int (*make_private)(struct inode *inode, pgoff_t start, pgoff_t end);
+	int (*make_shared)(struct inode *inode, pgoff_t start, pgoff_t end);
+	/* Prepare private RAM state after lookup, before returning a guest PFN. */
+	int (*prepare_private_pfn)(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+				   kvm_pfn_t nr_pages);
+};
+
 struct gmem_inode {
 	struct shared_policy policy;
 	struct inode vfs_inode;
@@ -40,6 +50,7 @@ struct gmem_inode {
 
 	void *provider;
 	const struct guest_memfd_provider_operations *provider_ops;
+	const struct kvm_gmem_backing_ops *backing_ops;
 
 	u64 flags;
 	/*
@@ -51,6 +62,9 @@ struct gmem_inode {
 	struct maple_tree attributes;
 };
 
+static const struct kvm_gmem_backing_ops *
+kvm_gmem_select_backing_ops(struct gmem_inode *gi);
+
 static __always_inline struct gmem_inode *GMEM_I(struct inode *inode)
 {
 	return container_of(inode, struct gmem_inode, vfs_inode);
@@ -103,6 +117,7 @@ static bool gmem_provider_ops_valid(struct gmem_inode *gi,
 
 static int kvm_gmem_attach_resource(struct inode *inode, int resource_fd)
 {
+	struct gmem_inode *gi = GMEM_I(inode);
 	const struct guest_memfd_provider_operations *ops = NULL;
 	void *provider = ERR_PTR(-EOPNOTSUPP);
 	struct file *resource_file = NULL;
@@ -118,7 +133,7 @@ static int kvm_gmem_attach_resource(struct inode *inode, int resource_fd)
 			ops = resource_file->f_op->gmem_provider_ops;
 	}
 
-	if (gmem_provider_ops_valid(GMEM_I(inode), ops))
+	if (gmem_provider_ops_valid(gi, ops))
 		provider = ops->attach(inode, resource_file);
 
 	if (resource_file)
@@ -127,8 +142,9 @@ static int kvm_gmem_attach_resource(struct inode *inode, int resource_fd)
 	if (IS_ERR(provider))
 		return PTR_ERR(provider);
 
-	GMEM_I(inode)->provider = provider;
-	GMEM_I(inode)->provider_ops = ops;
+	gi->provider = provider;
+	gi->provider_ops = ops;
+	gi->backing_ops = kvm_gmem_select_backing_ops(gi);
 
 	return 0;
 }
@@ -918,31 +934,15 @@ static bool kvm_gmem_has_outstanding_references(struct inode *inode,
 }
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
-static void kvm_gmem_make_shared(struct inode *inode, pgoff_t start, pgoff_t end)
+static int kvm_gmem_folio_make_shared(struct inode *inode,
+		pgoff_t start, pgoff_t end)
 {
-	struct gmem_inode *gi = GMEM_I(inode);
 	struct folio_batch fbatch;
 	pgoff_t next = start;
 	int i;
 
-	if (kvm_gmem_is_pfn_mode(gi)) {
-		pgoff_t index;
-
-		for (index = start; index < end; ) {
-			size_t step = 1;
-			size_t nr_pages;
-			kvm_pfn_t pfn;
-
-			if (!gmem_provider_lookup_pfn_range(gi, index, &pfn, &nr_pages)) {
-				step = min_t(size_t, nr_pages, end - index);
-				kvm_arch_gmem_make_shared(pfn, step);
-			}
-
-			index += step;
-			cond_resched();
-		}
-		return;
-	}
+	if (!kvm_arch_has_gmem_convert())
+		return 0;
 
 	folio_batch_init(&fbatch);
 	while (filemap_get_folios(inode->i_mapping, &next, end - 1, &fbatch)) {
@@ -969,11 +969,128 @@ static void kvm_gmem_make_shared(struct inode *inode, pgoff_t start, pgoff_t end
 		folio_batch_release(&fbatch);
 		cond_resched();
 	}
+
+	return 0;
+}
+
+static int kvm_gmem_ram_pfn_make_shared(struct inode *inode,
+		pgoff_t start, pgoff_t end)
+{
+	struct gmem_inode *gi = GMEM_I(inode);
+	pgoff_t index;
+
+	if (!kvm_arch_has_gmem_convert())
+		return 0;
+
+	for (index = start; index < end; ) {
+		size_t step = 1;
+		size_t nr_pages;
+		kvm_pfn_t pfn;
+
+		if (!gmem_provider_lookup_pfn_range(gi, index, &pfn, &nr_pages)) {
+			step = min_t(size_t, nr_pages, end - index);
+			kvm_arch_gmem_make_shared(pfn, step);
+		}
+
+		index += step;
+		cond_resched();
+	}
+
+	return 0;
 }
 #else
-static void kvm_gmem_make_shared(struct inode *inode, pgoff_t start, pgoff_t end) {}
+static int kvm_gmem_folio_make_shared(struct inode *inode,
+		pgoff_t start, pgoff_t end)
+{
+	return 0;
+}
+
+static int kvm_gmem_ram_pfn_make_shared(struct inode *inode,
+		pgoff_t start, pgoff_t end)
+{
+	return 0;
+}
 #endif
 
+static int kvm_gmem_folio_prepare_attributes(struct inode *inode, pgoff_t start,
+		size_t nr_pages, u64 attrs, pgoff_t *err_index)
+{
+	if (!(attrs & KVM_MEMORY_ATTRIBUTE_PRIVATE))
+		return 0;
+
+	/*
+	 * Forcefully unmap the pages from all userspace page tables,
+	 * and then verify there are no outstanding references, e.g.
+	 * acquired via GUP or similar.  Tell userspace to try again if
+	 * there are outstanding references and hope that whatever has
+	 * pinned the page will put its reference "soon".
+	 */
+	unmap_mapping_pages(inode->i_mapping, start, nr_pages, false);
+	if (kvm_gmem_has_outstanding_references(inode, start, nr_pages, err_index))
+		return -EAGAIN;
+
+	return 0;
+}
+
+static int kvm_gmem_ram_pfn_prepare_attributes(struct inode *inode, pgoff_t start,
+		size_t nr_pages, u64 attrs, pgoff_t *err_index)
+{
+	if (!(attrs & KVM_MEMORY_ATTRIBUTE_PRIVATE))
+		return 0;
+
+	unmap_mapping_pages(inode->i_mapping, start, nr_pages, false);
+	return 0;
+}
+
+static int kvm_gmem_noop_conversion(struct inode *inode, pgoff_t start, pgoff_t end)
+{
+	/* Mapping changes are completed by invalidation or the fault path. */
+	return 0;
+}
+
+static int kvm_gmem_ram_prepare_private_pfn(struct kvm *kvm, gfn_t gfn,
+					    kvm_pfn_t pfn, kvm_pfn_t nr_pages)
+{
+	if (kvm_arch_has_gmem_convert())
+		return kvm_arch_gmem_make_private(kvm, gfn, pfn, nr_pages);
+
+	return 0;
+}
+
+static const struct kvm_gmem_backing_ops kvm_gmem_folio_backing_ops = {
+	.prepare_attributes = kvm_gmem_folio_prepare_attributes,
+	.make_private = kvm_gmem_noop_conversion,
+	.make_shared = kvm_gmem_folio_make_shared,
+	.prepare_private_pfn = kvm_gmem_ram_prepare_private_pfn,
+};
+
+static const struct kvm_gmem_backing_ops kvm_gmem_ram_pfn_backing_ops = {
+	.prepare_attributes = kvm_gmem_ram_pfn_prepare_attributes,
+	.make_private = kvm_gmem_noop_conversion,
+	.make_shared = kvm_gmem_ram_pfn_make_shared,
+	.prepare_private_pfn = kvm_gmem_ram_prepare_private_pfn,
+};
+
+static const struct kvm_gmem_backing_ops *
+kvm_gmem_select_backing_ops(struct gmem_inode *gi)
+{
+	if (kvm_gmem_is_pfn_mode(gi))
+		return &kvm_gmem_ram_pfn_backing_ops;
+
+	return &kvm_gmem_folio_backing_ops;
+}
+
+static int kvm_gmem_convert_attributes(struct inode *inode, pgoff_t start,
+		pgoff_t end, u64 attrs)
+{
+	const struct kvm_gmem_backing_ops *ops = GMEM_I(inode)->backing_ops;
+
+	if (attrs & KVM_MEMORY_ATTRIBUTE_PRIVATE)
+		return ops->make_private(inode, start, end);
+
+	return ops->make_shared(inode, start, end);
+}
+
 static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 				     size_t nr_pages, uint64_t attrs,
 				     pgoff_t *err_index)
@@ -983,57 +1100,39 @@ static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 	struct gmem_inode *gi = GMEM_I(inode);
 	enum kvm_gfn_range_filter filter;
 	pgoff_t end = start + nr_pages;
-	struct maple_tree *mt;
 	struct ma_state mas;
 	int r = 0;
 
-	mt = &gi->attributes;
-
+	*err_index = start;
 	filemap_invalidate_lock(mapping);
 
 	if (__kvm_gmem_range_has_attributes(inode, start, nr_pages, attrs))
 		goto out;
 
-	mas_init(&mas, mt, start);
+	mas_init(&mas, &gi->attributes, start);
 	r = kvm_gmem_mas_preallocate(&mas, attrs, start, nr_pages);
-	if (r) {
-		*err_index = start;
+	if (r)
 		goto out;
-	}
 
-	if (to_private) {
-		/*
-		 * Forcefully unmap the pages from all userspace page tables,
-		 * and then verify there are no outstanding references, e.g.
-		 * acquired via GUP or similar.  Tell userspace to try again if
-		 * there are outstanding references and hope that whatever has
-		 * pinned the page will put its reference "soon".
-		 */
-		unmap_mapping_pages(mapping, start, nr_pages, false);
-
-		if (!kvm_gmem_is_pfn_mode(gi) &&
-		    kvm_gmem_has_outstanding_references(inode, start, nr_pages,
-							err_index)) {
-			mas_destroy(&mas);
-			r = -EAGAIN;
-			goto out;
-		}
-	}
-
-	/*
-	 * From this point on guest_memfd has performed necessary
-	 * checks and can proceed to do guest-breaking changes.
-	 */
+	r = gi->backing_ops->prepare_attributes(inode, start, nr_pages, attrs,
+						  err_index);
+	if (r)
+		goto destroy;
 
 	filter = to_private ? KVM_FILTER_SHARED : KVM_FILTER_PRIVATE;
 	kvm_gmem_invalidate_start(inode, start, end, filter);
 
-	if (!to_private && kvm_arch_has_gmem_convert())
-		kvm_gmem_make_shared(inode, start, end);
-
-	mas_store_prealloc(&mas, xa_mk_value(attrs));
+	r = kvm_gmem_convert_attributes(inode, start, end, attrs);
+	if (!r)
+		mas_store_prealloc(&mas, xa_mk_value(attrs));
 
+	if (r)
+		mas_destroy(&mas);
 	kvm_gmem_invalidate_end(inode, start, end);
+	goto out;
+
+destroy:
+	mas_destroy(&mas);
 out:
 	filemap_invalidate_unlock(mapping);
 	return r;
@@ -1559,6 +1658,7 @@ static struct folio *__kvm_gmem_get_pfn(struct file *file,
 int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 		     gfn_t gfn, kvm_pfn_t *pfn, int *max_order)
 {
+	const struct kvm_gmem_backing_ops *ops;
 	pgoff_t index = kvm_gmem_get_index(slot, gfn);
 	struct folio *folio;
 	int r = 0, __order;
@@ -1569,6 +1669,8 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 	if (!file)
 		return -EFAULT;
 
+	ops = GMEM_I(file_inode(file))->backing_ops;
+
 	filemap_invalidate_lock_shared(file_inode(file)->i_mapping);
 
 	folio = __kvm_gmem_get_pfn(file, slot, index, pfn, max_order);
@@ -1577,10 +1679,10 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 		goto out;
 	}
 
-	if (kvm_arch_has_gmem_convert() &&
+	if (ops->prepare_private_pfn &&
 	    kvm_gmem_is_private_mem(file_inode(file), index))
-		r = kvm_arch_gmem_make_private(kvm, gfn, *pfn,
-					       (kvm_pfn_t)1 << *max_order);
+		r = ops->prepare_private_pfn(kvm, gfn, *pfn,
+					   (kvm_pfn_t)1 << *max_order);
 
 	if (folio) {
 		folio_unlock(folio);
@@ -1729,6 +1831,7 @@ static struct inode *kvm_gmem_alloc_inode(struct super_block *sb)
 	gi->flags = 0;
 	gi->provider = NULL;
 	gi->provider_ops = NULL;
+	gi->backing_ops = NULL;
 	INIT_LIST_HEAD(&gi->gmem_file_list);
 	return &gi->vfs_inode;
 }
-- 
2.43.0

[RFC PATCH v3 03/13] KVM: guest_memfd: Let providers coordinate mapping changes

[Aneesh Kumar K.V (Arm)]

Resource providers may need to serialize host access with guest
invalidation, architecture conversion and publication of new attributes.

Add optional paired lock() and unlock() callbacks without exposing KVM
memslots or architecture requests through the provider interface. Reject
providers that supply only one callback.

Acquire provider exclusion after the inode invalidate lock and before
guest invalidation starts. Release it after invalidation ends and any
successful attribute publication, including when conversion fails.
These hooks only provide exclusion and cannot reject cleanup. Subsequent
revocation, memslot unbind and final file cleanup use the same pair
without changing attributes.

Existing providers omit both callbacks and retain their current behavior.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 include/linux/guest_memfd.h | 11 +++++++++++
 virt/kvm/guest_memfd.c      | 22 ++++++++++++++++++++--
 2 files changed, 31 insertions(+), 2 deletions(-)

diff --git a/include/linux/guest_memfd.h b/include/linux/guest_memfd.h
index 17407504307b..9fc04f1188c1 100644
--- a/include/linux/guest_memfd.h
+++ b/include/linux/guest_memfd.h
@@ -20,6 +20,15 @@ struct mempolicy;
  *                         Offset and size are in bytes. The flags contain only
  *                         KVM_MEM_LOG_DIRTY_PAGES and KVM_MEM_READONLY bits.
  *                         Return 0 on success or a negative error code.
+ * @lock: Optional provider exclusion for mapping and attribute changes. Called
+ *        with the inode invalidate lock held for write, before guest
+ *        invalidation starts. May sleep and must return with exclusion held.
+ *        Supply both lock and unlock, or neither.
+ * @unlock: Release exclusion acquired by lock(). Called after guest
+ *          invalidation ends and any successful attribute publication, with
+ *          the inode invalidate lock still held. The pair also covers range
+ *          revocation, memslot unbind and final file cleanup; those operations
+ *          do not change attributes.
  * @alloc_folio: Allocate an uninserted folio for a given index and NUMA policy.
  *               Must return a pointer to a struct folio on success, or an
  *               ERR_PTR() error code on failure. Must never return NULL.
@@ -52,6 +61,8 @@ struct guest_memfd_provider_operations {
 	void (*release)(void *provider);
 	int (*prepare_memory_region)(void *provider, u64 offset, u64 size,
 				     u32 flags);
+	void (*lock)(void *provider);
+	void (*unlock)(void *provider);
 	struct folio *(*alloc_folio)(void *provider, pgoff_t index,
 				     struct mempolicy *mpol);
 	void (*invalidate_folio)(void *provider, struct folio *folio);
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index bd8324b4baba..d1060b4c785f 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -107,6 +107,9 @@ static bool gmem_provider_ops_valid(struct gmem_inode *gi,
 	if (!ops || !ops->attach)
 		return false;
 
+	if (!!ops->lock != !!ops->unlock)
+		return false;
+
 	if (kvm_gmem_is_pfn_mode(gi))
 		return ops->get_pfn && ops->lookup_pfn_range;
 
@@ -208,6 +211,18 @@ static void gmem_provider_put_pfn_range(struct gmem_inode *gi, pgoff_t index,
 		gi->provider_ops->put_pfn_range(gi->provider, index, nr_pages);
 }
 
+static void gmem_provider_lock(struct gmem_inode *gi)
+{
+	if (gi->provider_ops && gi->provider_ops->lock)
+		gi->provider_ops->lock(gi->provider);
+}
+
+static void gmem_provider_unlock(struct gmem_inode *gi)
+{
+	if (gi->provider_ops && gi->provider_ops->unlock)
+		gi->provider_ops->unlock(gi->provider);
+}
+
 static void gmem_provider_release(struct gmem_inode *gi)
 {
 	if (gi->provider_ops && gi->provider_ops->release)
@@ -1092,8 +1107,7 @@ static int kvm_gmem_convert_attributes(struct inode *inode, pgoff_t start,
 }
 
 static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
-				     size_t nr_pages, uint64_t attrs,
-				     pgoff_t *err_index)
+		size_t nr_pages, u64 attrs, pgoff_t *err_index)
 {
 	bool to_private = attrs & KVM_MEMORY_ATTRIBUTE_PRIVATE;
 	struct address_space *mapping = inode->i_mapping;
@@ -1119,6 +1133,9 @@ static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 	if (r)
 		goto destroy;
 
+	/* Provider exclusion must cover invalidation as well as conversion. */
+	gmem_provider_lock(gi);
+
 	filter = to_private ? KVM_FILTER_SHARED : KVM_FILTER_PRIVATE;
 	kvm_gmem_invalidate_start(inode, start, end, filter);
 
@@ -1129,6 +1146,7 @@ static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 	if (r)
 		mas_destroy(&mas);
 	kvm_gmem_invalidate_end(inode, start, end);
+	gmem_provider_unlock(gi);
 	goto out;
 
 destroy:
-- 
2.43.0

[RFC PATCH v3 04/13] KVM: guest_memfd: Dispatch revocation through backing operations

[Aneesh Kumar K.V (Arm)]

Revocation truncates cached folios for folio backing, while RAM PFN
backing unmaps host mappings, reclaims architecture state and optionally
releases provider PFNs. Move that into backing-specific revoke()
operations.

Hold optional provider exclusion across invalidation and revocation,
inside the inode invalidate lock. Keep reclamation and provider PFN
release in the backing operation rather than the common sequence.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 virt/kvm/guest_memfd.c | 62 ++++++++++++++++++++++++++++++------------
 1 file changed, 44 insertions(+), 18 deletions(-)

diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index d1060b4c785f..6c91fec0259e 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -34,6 +34,8 @@ struct gmem_file {
 static inline struct file *kvm_gmem_get_file(struct kvm_memory_slot *slot);
 
 struct kvm_gmem_backing_ops {
+	int (*revoke)(struct inode *inode, pgoff_t start, pgoff_t end,
+		const unsigned long *pfn, bool put_pfn_range);
 	int (*prepare_attributes)(struct inode *inode, pgoff_t start,
 			size_t nr_pages, u64 attrs, pgoff_t *err_index);
 	int (*make_private)(struct inode *inode, pgoff_t start, pgoff_t end);
@@ -481,14 +483,48 @@ static void kvm_gmem_pfn_reclaim(struct gmem_inode *gi, pgoff_t start, pgoff_t e
 static void kvm_gmem_pfn_reclaim(struct gmem_inode *gi, pgoff_t start, pgoff_t end) {}
 #endif
 
+static int kvm_gmem_folio_revoke(struct inode *inode, pgoff_t start, pgoff_t end,
+		const unsigned long *pfn, bool put_pfn_range)
+{
+	loff_t offset = (loff_t)start << PAGE_SHIFT;
+	loff_t len = (loff_t)(end - start) << PAGE_SHIFT;
+
+	truncate_inode_pages_range(inode->i_mapping, offset, offset + len - 1);
+	return 0;
+}
+
+static int kvm_gmem_ram_pfn_revoke(struct inode *inode, pgoff_t start, pgoff_t end,
+		const unsigned long *pfn, bool put_pfn_range)
+{
+	loff_t offset = (loff_t)start << PAGE_SHIFT;
+	loff_t len = (loff_t)(end - start) << PAGE_SHIFT;
+	size_t nr_pages = end - start;
+
+	unmap_mapping_range(inode->i_mapping, offset, len, 0);
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
+	if (pfn)
+		kvm_arch_gmem_reclaim(*pfn, nr_pages);
+	else
+		kvm_gmem_pfn_reclaim(GMEM_I(inode), start, end);
+#endif
+	if (put_pfn_range)
+		gmem_provider_put_pfn_range(GMEM_I(inode), start, nr_pages);
+	return 0;
+}
+
 static int __kvm_gmem_revoke(struct inode *inode, pgoff_t start,
 			      size_t nr_pages, const unsigned long *pfn,
 			      bool put_pfn_range)
 {
 	enum kvm_gfn_range_filter filter = kvm_gmem_get_all_gfns_filter(inode);
 	pgoff_t max_pages = i_size_read(inode) >> PAGE_SHIFT;
-	loff_t offset, len;
+	struct gmem_inode *gi = GMEM_I(inode);
+	const struct kvm_gmem_backing_ops *ops = gi->backing_ops;
 	pgoff_t end;
+	int r;
+
+	if (!ops->revoke)
+		return -EOPNOTSUPP;
 
 	if (start >= max_pages)
 		return 0;
@@ -497,8 +533,6 @@ static int __kvm_gmem_revoke(struct inode *inode, pgoff_t start,
 	if (!nr_pages)
 		return 0;
 
-	offset = (loff_t)start << PAGE_SHIFT;
-	len = (loff_t)nr_pages << PAGE_SHIFT;
 	end = start + nr_pages;
 
 	/*
@@ -507,27 +541,17 @@ static int __kvm_gmem_revoke(struct inode *inode, pgoff_t start,
 	 */
 	filemap_invalidate_lock(inode->i_mapping);
 
+	gmem_provider_lock(gi);
+
 	kvm_gmem_invalidate_start(inode, start, end, filter);
 
-	if (kvm_gmem_is_pfn_mode(GMEM_I(inode))) {
-		unmap_mapping_range(inode->i_mapping, offset, len, 0);
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
-		if (pfn)
-			kvm_arch_gmem_reclaim(*pfn, nr_pages);
-		else
-			kvm_gmem_pfn_reclaim(GMEM_I(inode), start, end);
-#endif
-		if (put_pfn_range)
-			gmem_provider_put_pfn_range(GMEM_I(inode), start, nr_pages);
-	} else {
-		truncate_inode_pages_range(inode->i_mapping, offset, offset + len - 1);
-	}
+	r = ops->revoke(inode, start, end, pfn, put_pfn_range);
 
 	kvm_gmem_invalidate_end(inode, start, end);
-
+	gmem_provider_unlock(gi);
 	filemap_invalidate_unlock(inode->i_mapping);
 
-	return 0;
+	return r;
 }
 
 static long kvm_gmem_punch_hole(struct inode *inode, loff_t offset, loff_t len)
@@ -1073,6 +1097,7 @@ static int kvm_gmem_ram_prepare_private_pfn(struct kvm *kvm, gfn_t gfn,
 }
 
 static const struct kvm_gmem_backing_ops kvm_gmem_folio_backing_ops = {
+	.revoke = kvm_gmem_folio_revoke,
 	.prepare_attributes = kvm_gmem_folio_prepare_attributes,
 	.make_private = kvm_gmem_noop_conversion,
 	.make_shared = kvm_gmem_folio_make_shared,
@@ -1080,6 +1105,7 @@ static const struct kvm_gmem_backing_ops kvm_gmem_folio_backing_ops = {
 };
 
 static const struct kvm_gmem_backing_ops kvm_gmem_ram_pfn_backing_ops = {
+	.revoke = kvm_gmem_ram_pfn_revoke,
 	.prepare_attributes = kvm_gmem_ram_pfn_prepare_attributes,
 	.make_private = kvm_gmem_noop_conversion,
 	.make_shared = kvm_gmem_ram_pfn_make_shared,
-- 
2.43.0

[RFC PATCH v3 05/13] KVM: guest_memfd: Split population by backing type

[Aneesh Kumar K.V (Arm)]

Preallocation obtains cached folios for folio backing and walks provider
PFN ranges for RAM PFN backing. Both use the same range validation and
shared inode invalidate lock.

Move the two walks into backing-specific populate() operations. Keep
size checks and locking in kvm_gmem_allocate(), preserving signal
handling, folio references and PFN range stepping.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 virt/kvm/guest_memfd.c | 101 +++++++++++++++++++++++++----------------
 1 file changed, 61 insertions(+), 40 deletions(-)

diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 6c91fec0259e..4c2fe151c33d 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -34,6 +34,7 @@ struct gmem_file {
 static inline struct file *kvm_gmem_get_file(struct kvm_memory_slot *slot);
 
 struct kvm_gmem_backing_ops {
+	int (*populate)(struct inode *inode, pgoff_t start, pgoff_t end);
 	int (*revoke)(struct inode *inode, pgoff_t start, pgoff_t end,
 		const unsigned long *pfn, bool put_pfn_range);
 	int (*prepare_attributes)(struct inode *inode, pgoff_t start,
@@ -586,46 +587,11 @@ int kvm_gmem_revoke(struct inode *inode, pgoff_t start, size_t nr_pages,
 }
 EXPORT_SYMBOL_GPL(kvm_gmem_revoke);
 
-static long kvm_gmem_allocate(struct inode *inode, loff_t offset, loff_t len)
+static int kvm_gmem_folio_populate(struct inode *inode,
+		pgoff_t start, pgoff_t end)
 {
-	struct address_space *mapping = inode->i_mapping;
-	struct gmem_inode *gi = GMEM_I(inode);
-	pgoff_t start, index, end;
-	int r;
-
-	/* Dedicated guest is immutable by default. */
-	if (offset + len > i_size_read(inode))
-		return -EINVAL;
-
-	filemap_invalidate_lock_shared(mapping);
-
-	start = offset >> PAGE_SHIFT;
-	end = (offset + len) >> PAGE_SHIFT;
-
-	r = 0;
-	if (kvm_gmem_is_pfn_mode(gi)) {
-		for (index = start; index < end; ) {
-			pgoff_t next_boundary;
-			pgoff_t max_nr_pages;
-			int max_order = 0;
-			kvm_pfn_t pfn;
-
-			if (signal_pending(current)) {
-				r = -EINTR;
-				break;
-			}
-
-			r = gmem_provider_get_pfn(gi, index, &pfn, &max_order);
-			if (r)
-				break;
-
-			max_nr_pages = 1UL << max_order;
-			next_boundary = ALIGN_DOWN(index + max_nr_pages, max_nr_pages);
-			index += min_t(size_t, next_boundary - index, end - index);
-			cond_resched();
-		}
-		goto out_unlock;
-	}
+	pgoff_t index;
+	int r = 0;
 
 	for (index = start; index < end; ) {
 		struct folio *folio;
@@ -653,7 +619,60 @@ static long kvm_gmem_allocate(struct inode *inode, loff_t offset, loff_t len)
 		cond_resched();
 	}
 
-out_unlock:
+	return r;
+}
+
+static int kvm_gmem_ram_pfn_populate(struct inode *inode,
+		pgoff_t start, pgoff_t end)
+{
+	struct gmem_inode *gi = GMEM_I(inode);
+	pgoff_t index;
+	int r = 0;
+
+	for (index = start; index < end; ) {
+		pgoff_t next_boundary;
+		pgoff_t max_nr_pages;
+		int max_order = 0;
+		kvm_pfn_t pfn;
+
+		if (signal_pending(current)) {
+			r = -EINTR;
+			break;
+		}
+
+		r = gmem_provider_get_pfn(gi, index, &pfn, &max_order);
+		if (r)
+			break;
+
+		max_nr_pages = 1UL << max_order;
+		next_boundary = ALIGN_DOWN(index + max_nr_pages, max_nr_pages);
+		index += min_t(size_t, next_boundary - index, end - index);
+		cond_resched();
+	}
+
+	return r;
+}
+
+static long kvm_gmem_allocate(struct inode *inode, loff_t offset, loff_t len)
+{
+	struct address_space *mapping = inode->i_mapping;
+	const struct kvm_gmem_backing_ops *ops = GMEM_I(inode)->backing_ops;
+	pgoff_t start, end;
+	int r;
+
+	if (!ops->populate)
+		return -EOPNOTSUPP;
+
+	/* Dedicated guest is immutable by default. */
+	if (offset + len > i_size_read(inode))
+		return -EINVAL;
+
+	filemap_invalidate_lock_shared(mapping);
+
+	start = offset >> PAGE_SHIFT;
+	end = (offset + len) >> PAGE_SHIFT;
+	r = ops->populate(inode, start, end);
+
 	filemap_invalidate_unlock_shared(mapping);
 
 	return r;
@@ -1097,6 +1116,7 @@ static int kvm_gmem_ram_prepare_private_pfn(struct kvm *kvm, gfn_t gfn,
 }
 
 static const struct kvm_gmem_backing_ops kvm_gmem_folio_backing_ops = {
+	.populate = kvm_gmem_folio_populate,
 	.revoke = kvm_gmem_folio_revoke,
 	.prepare_attributes = kvm_gmem_folio_prepare_attributes,
 	.make_private = kvm_gmem_noop_conversion,
@@ -1105,6 +1125,7 @@ static const struct kvm_gmem_backing_ops kvm_gmem_folio_backing_ops = {
 };
 
 static const struct kvm_gmem_backing_ops kvm_gmem_ram_pfn_backing_ops = {
+	.populate = kvm_gmem_ram_pfn_populate,
 	.revoke = kvm_gmem_ram_pfn_revoke,
 	.prepare_attributes = kvm_gmem_ram_pfn_prepare_attributes,
 	.make_private = kvm_gmem_noop_conversion,
-- 
2.43.0

[RFC PATCH v3 06/13] KVM: guest_memfd: Support device-backed PFN providers

[Aneesh Kumar K.V (Arm)]

Device resources such as PCI BARs supply PFNs directly and cannot use
the folio allocation path. Store the mapping owner and device ID in the
core context during attach(), and use them when preparing device
memslots. Keep device memory separate from ordinary PFN-backed RAM.

Require device providers to use PFN_MODE and INIT_SHARED with
architecture support for protected device mappings. Device providers
require get_pfn(), but not the lookup_pfn_range() callback.

Mark device memslots as guest_memfd-only and pass their device binding
to the architecture. Resolve guest PFN lookup through get_pfn() for
either shared or private attributes; the architecture decides how to
establish mappings. Include both mapping types in generic invalidation,
using the memslot binding to select the architecture unmap path. Host
userspace access still requires shared attributes.

Skip RAM population and PFN reclamation for device MMIO, which has no
RAM pages to initialize or reclaim.

Keep mmap, fallocate and attribute changes on device-backed guest_memfd
files disabled until device mapping and conversion support is added.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 Documentation/virt/kvm/api.rst | 16 ++++--
 include/linux/guest_memfd.h    | 53 ++++++++++++++++++--
 include/linux/kvm_host.h       |  5 ++
 virt/kvm/guest_memfd.c         | 89 +++++++++++++++++++++++++++++++---
 4 files changed, 146 insertions(+), 17 deletions(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index baa88f8b4cc7..a07c04a3890d 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -6481,8 +6481,10 @@ that refers to it.  guest_memfd files are roughly analogous to files created
 via memfd_create(), e.g. guest_memfd files live in RAM, have volatile storage,
 and are automatically released when the last reference is dropped.  Unlike
 "regular" memfd_create() files, guest_memfd files are bound to their owning
-virtual machine (see below), cannot be mapped, read, or written by userspace,
-and cannot be resized  (guest_memfd files do however support PUNCH_HOLE).
+virtual machine (see below), cannot be read or written by userspace, and
+cannot be resized. Userspace mappings depend on the creation flags and backing
+resource. Ordinary guest_memfd files support PUNCH_HOLE; device-backed files
+reject fallocate().
 
 ::
 
@@ -6531,10 +6533,16 @@ specified via KVM_CREATE_GUEST_MEMFD.  Currently defined flags:
                                 by an external resource provider. When set,
                                 GUEST_MEMFD_FLAG_USE_RESOURCE must also be set,
                                 and the provider must support PFN lookup
-                                (.get_pfn and .lookup_pfn_range) rather than
-                                folio allocation (.alloc_folio).
+                                (.get_pfn and, for RAM providers,
+                                .lookup_pfn_range) rather than folio allocation
+                                (.alloc_folio).
   ============================= ================================================
 
+With USE_RESOURCE, the resource file's operations supply the guest_memfd
+provider. PFN_MODE selects the PFN callbacks; providers that supply device
+operations additionally require INIT_SHARED and a VM supporting protected
+device mappings.
+
 When the KVM MMU performs a PFN lookup to service a guest fault, the fault will
 always be consumed from guest_memfd, regardless of whether it is a shared or a
 private fault (unless in-place conversion is disabled and the backing
diff --git a/include/linux/guest_memfd.h b/include/linux/guest_memfd.h
index 9fc04f1188c1..8a93004aa9e7 100644
--- a/include/linux/guest_memfd.h
+++ b/include/linux/guest_memfd.h
@@ -8,12 +8,39 @@ struct file;
 struct folio;
 struct inode;
 struct mempolicy;
+struct kvm;
+
+/* Binding data stays valid only while the provider is active. */
+struct guest_memfd_device_binding {
+	/* Architecture-specific mapping owner, e.g. an RMM vDEVICE handle. */
+	unsigned long owner;
+	u64 vdev_id;
+};
+
+/**
+ * struct guest_memfd_device - Device memory context supplied during attach()
+ * @kvm: VM owning the device mappings
+ * @size: File size in bytes
+ * @flags: GUEST_MEMFD_FLAG_* creation flags
+ * @binding: Mapping owner and device ID set by attach(), with the underlying
+ *           resources pinned until release()
+ */
+struct guest_memfd_device {
+	struct kvm *kvm;
+	u64 size;
+	u64 flags;
+	struct guest_memfd_device_binding binding;
+};
 
 /**
  * struct guest_memfd_provider_operations - Operations for external memory providers
  * @attach: Attach to a resource file, perform filesystem-specific validation,
- *          and return an opaque provider context.
- * @release: Release provider context and unpin any resources.
+ *          and return an opaque provider context. Pin resources and callback
+ *          code until release(). Device providers obtain their core context
+ *          through kvm_gmem_get_device().
+ * @release: Release provider context and unpin any resources. Must not access
+ *           the VM here; its file reference may be gone.
+ * @device_memory: Backing is device MMIO rather than RAM.
  * @prepare_memory_region: Optional check of a memory range and preparation of
  *                         access to its backing memory. Called after common
  *                         file checks and before publishing the memslot.
@@ -38,8 +65,12 @@ struct mempolicy;
  *           zeroing if necessary. On success, return 0 with the physical frame
  *           number in @pfn and, if @max_order is non-NULL, the maximum page
  *           order of contiguous backing memory containing @pfn. Returns a
- *           negative error code on failure. The provider must define
- *           @lookup_pfn_range when @get_pfn is defined.
+ *           negative error code on failure. RAM PFN providers must also define
+ *           @lookup_pfn_range. Device providers need only @get_pfn and are
+ *           called with the inode invalidate lock held, for read on faults.
+ *           Guest PFN lookup is allowed for either shared or private file
+ *           attributes; the architecture determines how mappings are created.
+ *           Host userspace faults require shared attributes.
  * @lookup_pfn_range: Look up the PFN for a given index. Returns 0 on success
  *                    with the physical frame number in @pfn and the number of
  *                    contiguous pages the mapping is valid for in @nr_pages.
@@ -47,7 +78,8 @@ struct mempolicy;
  *                    is present. If backing memory is not available, return
  *                    -ENOENT. Return a negative error code on other failure.
  *                    The provider must define @get_pfn when @lookup_pfn_range
- *                    is defined.
+ *                    is defined. Not used for device memory, whose protected
+ *                    mappings are removed through architecture device unmap.
  * @put_pfn_range: Release a range of PFNs previously obtained for a given
  *                 index.  The provider does not need to ensure that backing
  *                 memory is present. If backing memory is not available, the
@@ -71,12 +103,23 @@ struct guest_memfd_provider_operations {
 	int (*lookup_pfn_range)(void *provider, pgoff_t index,
 				unsigned long *pfn, size_t *nr_pages);
 	void (*put_pfn_range)(void *provider, pgoff_t index, size_t nr_pages);
+	bool device_memory;
 };
 
 #ifdef CONFIG_KVM_GUEST_MEMFD
+/*
+ * Call from device-provider attach(). Storage remains valid until release(),
+ * but the VM is valid only during attach() and core-initiated callbacks.
+ */
+struct guest_memfd_device *kvm_gmem_get_device(struct inode *inode);
 int kvm_gmem_revoke(struct inode *inode, pgoff_t start, size_t nr_pages,
 		    const unsigned long *pfn);
 #else
+static inline struct guest_memfd_device *kvm_gmem_get_device(struct inode *inode)
+{
+	return NULL;
+}
+
 static inline int kvm_gmem_revoke(struct inode *inode, pgoff_t start, size_t nr_pages,
 				  const unsigned long *pfn)
 {
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index ecf673cdb889..73d6c739b094 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -56,6 +56,7 @@
  */
 #define KVM_MEMSLOT_INVALID			(1UL << 16)
 #define KVM_MEMSLOT_GMEM_ONLY			(1UL << 17)
+#define KVM_MEMSLOT_GMEM_DEVICE			(1UL << 18)
 
 /*
  * Bit 63 of the memslot generation number is an "update in-progress flag",
@@ -2605,6 +2606,10 @@ static inline bool kvm_is_private_gfn(struct kvm *kvm, gfn_t gfn)
 #endif /* kvm_arch_has_private_mem */
 
 #ifdef CONFIG_KVM_GUEST_MEMFD
+struct guest_memfd_device_binding;
+bool kvm_arch_gmem_device_supported(struct kvm *kvm);
+void kvm_arch_gmem_device_bind(struct kvm_memory_slot *slot,
+		const struct guest_memfd_device_binding *binding);
 bool kvm_gmem_is_private_gfn(struct kvm *kvm, gfn_t gfn);
 bool kvm_gmem_range_has_attributes(struct kvm_memory_slot *slot, gfn_t start,
 				   gfn_t end, u64 attributes);
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 4c2fe151c33d..5215bdf8941b 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -55,6 +55,8 @@ struct gmem_inode {
 	const struct guest_memfd_provider_operations *provider_ops;
 	const struct kvm_gmem_backing_ops *backing_ops;
 
+	struct guest_memfd_device device;
+
 	u64 flags;
 	/*
 	 * Every index in this inode, whether memory is populated or
@@ -73,6 +75,19 @@ static __always_inline struct gmem_inode *GMEM_I(struct inode *inode)
 	return container_of(inode, struct gmem_inode, vfs_inode);
 }
 
+static bool gmem_provider_is_device(struct gmem_inode *gi)
+{
+	return gi->provider_ops && gi->provider_ops->device_memory;
+}
+
+struct guest_memfd_device *kvm_gmem_get_device(struct inode *inode)
+{
+	struct gmem_inode *gi = GMEM_I(inode);
+
+	return gi->device.kvm ? &gi->device : NULL;
+}
+EXPORT_SYMBOL_GPL(kvm_gmem_get_device);
+
 static bool kvm_gmem_is_pfn_mode(struct gmem_inode *gi)
 {
 	return gi->flags & GUEST_MEMFD_FLAG_PFN_MODE;
@@ -104,7 +119,7 @@ static const struct guest_memfd_provider_operations gmem_native_provider_ops = {
 	.alloc_folio = gmem_native_alloc_folio,
 };
 
-static bool gmem_provider_ops_valid(struct gmem_inode *gi,
+static bool gmem_provider_ops_valid(struct kvm *kvm, struct gmem_inode *gi,
 				     const struct guest_memfd_provider_operations *ops)
 {
 	if (!ops || !ops->attach)
@@ -113,16 +128,27 @@ static bool gmem_provider_ops_valid(struct gmem_inode *gi,
 	if (!!ops->lock != !!ops->unlock)
 		return false;
 
+	if (ops->device_memory) {
+		if (!kvm_gmem_is_pfn_mode(gi) ||
+		    !(gi->flags & GUEST_MEMFD_FLAG_INIT_SHARED) ||
+		    !kvm_arch_gmem_device_supported(kvm) ||
+		    !ops->release || ops->alloc_folio || ops->invalidate_folio ||
+		    !ops->prepare_memory_region)
+			return false;
+	}
+
 	if (kvm_gmem_is_pfn_mode(gi))
-		return ops->get_pfn && ops->lookup_pfn_range;
+		return ops->get_pfn && (ops->device_memory || ops->lookup_pfn_range);
 
 	return ops->alloc_folio != NULL;
 }
 
 #define GMEM_RESOURCE_FD_NATIVE (-1)
 
-static int kvm_gmem_attach_resource(struct inode *inode, int resource_fd)
+static int kvm_gmem_attach_resource(struct kvm *kvm, struct file *file,
+		int resource_fd)
 {
+	struct inode *inode = file_inode(file);
 	struct gmem_inode *gi = GMEM_I(inode);
 	const struct guest_memfd_provider_operations *ops = NULL;
 	void *provider = ERR_PTR(-EOPNOTSUPP);
@@ -139,8 +165,14 @@ static int kvm_gmem_attach_resource(struct inode *inode, int resource_fd)
 			ops = resource_file->f_op->gmem_provider_ops;
 	}
 
-	if (gmem_provider_ops_valid(gi, ops))
+	if (gmem_provider_ops_valid(kvm, gi, ops)) {
+		if (ops->device_memory) {
+			gi->device.kvm = kvm;
+			gi->device.size = i_size_read(inode);
+			gi->device.flags = gi->flags;
+		}
 		provider = ops->attach(inode, resource_file);
+	}
 
 	if (resource_file)
 		fput(resource_file);
@@ -378,9 +410,15 @@ static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index)
 	return folio;
 }
 
+/*
+ * Select the mapping types that should be covered by generic guest_memfd
+ * invalidation. Device backing can have both shared and private mappings.
+ * For RAM, this depends on the file's initial attributes and whether in-place
+ * conversion allows the same file to back both shared and private mappings.
+ */
 static enum kvm_gfn_range_filter kvm_gmem_get_all_gfns_filter(struct inode *inode)
 {
-	if (gmem_in_place_conversion)
+	if (gmem_provider_is_device(GMEM_I(inode)) || gmem_in_place_conversion)
 		return KVM_FILTER_SHARED | KVM_FILTER_PRIVATE;
 
 	if (GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED)
@@ -683,6 +721,10 @@ static long kvm_gmem_fallocate(struct file *file, int mode, loff_t offset,
 {
 	int ret;
 
+	/* Device MMIO cannot be preallocated or discarded by hole punching. */
+	if (gmem_provider_is_device(GMEM_I(file_inode(file))))
+		return -EOPNOTSUPP;
+
 	if (!(mode & FALLOC_FL_KEEP_SIZE))
 		return -EOPNOTSUPP;
 
@@ -868,7 +910,8 @@ static int kvm_gmem_mmap(struct file *file, struct vm_area_struct *vma)
 {
 	struct gmem_inode *gi = GMEM_I(file_inode(file));
 
-	if (!kvm_gmem_supports_mmap(file_inode(file)))
+	if (gmem_provider_is_device(gi) ||
+	    !kvm_gmem_supports_mmap(file_inode(file)))
 		return -ENODEV;
 
 	if ((vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) !=
@@ -1431,6 +1474,16 @@ static int kvm_gmem_init_inode(struct inode *inode, loff_t size, u64 flags)
 	return r;
 }
 
+bool __weak kvm_arch_gmem_device_supported(struct kvm *kvm)
+{
+	return false;
+}
+
+void __weak kvm_arch_gmem_device_bind(struct kvm_memory_slot *slot,
+		const struct guest_memfd_device_binding *binding)
+{
+}
+
 static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags, int resource_fd)
 {
 	static const char *name = "[kvm-gmem]";
@@ -1486,7 +1539,7 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags, int resour
 	 */
 	smp_wmb();
 
-	err = kvm_gmem_attach_resource(inode, resource_fd);
+	err = kvm_gmem_attach_resource(kvm, file, resource_fd);
 	if (err) {
 		fput(file);
 		put_unused_fd(fd);
@@ -1580,6 +1633,16 @@ int kvm_gmem_prepare_memory_region(struct kvm *kvm, struct kvm_memory_slot *slot
 		}
 	}
 
+	/*
+	 * Device slots resolve both shared and private faults through guest_memfd.
+	 * Identify them as device memory and pass the provider's binding to the
+	 * architecture so it can associate mappings with the protected device.
+	 */
+	if (gmem_provider_is_device(gi)) {
+		slot->flags |= KVM_MEMSLOT_GMEM_ONLY | KVM_MEMSLOT_GMEM_DEVICE;
+		kvm_arch_gmem_device_bind(slot, &gi->device.binding);
+	}
+
 	/*
 	 * memslots of flag KVM_MEM_GUEST_MEMFD are immutable to change, so
 	 * kvm_gmem_bind() must occur on a new memslot.  Because the memslot
@@ -1770,6 +1833,10 @@ static long __kvm_gmem_populate(struct kvm *kvm, struct kvm_memory_slot *slot,
 	kvm_pfn_t pfn;
 	int ret;
 
+	/* Device MMIO cannot be initialized through the RAM population callback. */
+	if (slot->flags & KVM_MEMSLOT_GMEM_DEVICE)
+		return -EINVAL;
+
 	filemap_invalidate_lock(file->f_mapping);
 
 	folio = __kvm_gmem_get_pfn(file, slot, index, &pfn, NULL);
@@ -1897,6 +1964,7 @@ static struct inode *kvm_gmem_alloc_inode(struct super_block *sb)
 	gi->provider = NULL;
 	gi->provider_ops = NULL;
 	gi->backing_ops = NULL;
+	memset(&gi->device, 0, sizeof(gi->device));
 	INIT_LIST_HEAD(&gi->gmem_file_list);
 	return &gi->vfs_inode;
 }
@@ -1908,8 +1976,13 @@ static void kvm_gmem_destroy_inode(struct inode *inode)
 	/*
 	 * Skip if the provider was not yet set up, as iput() can reach this on
 	 * error paths before provider_ops is initialized.
+	 *
+	 * Device PFNs describe MMIO, with no RAM pages to reclaim or return to
+	 * the provider. Protected device mappings require architecture device
+	 * unmap instead of this RAM PFN cleanup.
 	 */
-	if (gi->provider_ops && kvm_gmem_is_pfn_mode(gi)) {
+	if (gi->provider_ops && kvm_gmem_is_pfn_mode(gi) &&
+	    !gmem_provider_is_device(gi)) {
 		pgoff_t nr_pages = i_size_read(inode) >> PAGE_SHIFT;
 
 		kvm_gmem_pfn_reclaim(gi, 0, nr_pages);
-- 
2.43.0

[RFC PATCH v3 07/13] KVM: guest_memfd: Support private/shared conversion of device memory

[Aneesh Kumar K.V (Arm)]

Allow device-backed guest_memfd ranges to be converted between shared
and private memory using file offsets. Coordinate conversion with the
device provider and
update guest_memfd attributes only after the architecture operation
succeeds.

Use the common attribute conversion path for RAM and device memory,
including inode locking, maple-tree allocation, guest invalidation and
attribute publication. Resolve guest addresses through the memslot bindings
and obtain physical addresses through the provider's get_pfn() callback.
Map only ranges whose attributes need to change, splitting at memslot and
physical-contiguity boundaries.

Reuse PFN attribute preparation for RAM and device memory to revoke host
mappings before private conversion. Supply device backing operations for
eager private mapping. The mapping operation resolves the required memslot
bindings. Shared conversion uses generic invalidation with the PRIVATE
filter before publishing attributes and needs no separate unmap callback.
Keep the architecture device mapping hook separate from RAM conversion.
Leave revoke() and populate() unset,
so full revocation, hole punching and preallocation remain unsupported.
The VMM validates guest MMIO requests against its BAR layout before setting
the corresponding guest_memfd range private. Architecture request
completion
validates the resulting mapping without installing it again.

Hold the inode invalidate lock across conversion and the attribute
update. Provider callbacks allow a device conversion lock to cover the
same operation, including guest invalidation, with the inode lock acquired
first. This prevents faults
and competing device operations from observing an incomplete conversion.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 Documentation/virt/kvm/api.rst |  5 ++
 include/linux/kvm_host.h       |  2 +
 virt/kvm/guest_memfd.c         | 83 ++++++++++++++++++++++++++++++++--
 3 files changed, 87 insertions(+), 3 deletions(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index a07c04a3890d..80359e927ff7 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -6543,6 +6543,11 @@ provider. PFN_MODE selects the PFN callbacks; providers that supply device
 operations additionally require INIT_SHARED and a VM supporting protected
 device mappings.
 
+KVM_SET_MEMORY_ATTRIBUTES2 converts device memory using guest_memfd offsets.
+For private conversion, KVM resolves the guest address through the memslot
+binding and obtains the physical address from the provider. Shared conversion
+removes the protected device mappings before publishing shared attributes.
+
 When the KVM MMU performs a PFN lookup to service a guest fault, the fault will
 always be consumed from guest_memfd, regardless of whether it is a shared or a
 private fault (unless in-place conversion is disabled and the backing
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 73d6c739b094..52b2664764c0 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -2610,6 +2610,8 @@ struct guest_memfd_device_binding;
 bool kvm_arch_gmem_device_supported(struct kvm *kvm);
 void kvm_arch_gmem_device_bind(struct kvm_memory_slot *slot,
 		const struct guest_memfd_device_binding *binding);
+int kvm_arch_gmem_device_map(struct kvm *kvm, const struct kvm_memory_slot *slot,
+			     u64 gpa, u64 size, u64 pa);
 bool kvm_gmem_is_private_gfn(struct kvm *kvm, gfn_t gfn);
 bool kvm_gmem_range_has_attributes(struct kvm_memory_slot *slot, gfn_t start,
 				   gfn_t end, u64 attributes);
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 5215bdf8941b..3bf26490f4e9 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -80,6 +80,8 @@ static bool gmem_provider_is_device(struct gmem_inode *gi)
 	return gi->provider_ops && gi->provider_ops->device_memory;
 }
 
+static int kvm_gmem_device_make_private(struct inode *inode, pgoff_t start, pgoff_t end);
+
 struct guest_memfd_device *kvm_gmem_get_device(struct inode *inode)
 {
 	struct gmem_inode *gi = GMEM_I(inode);
@@ -1133,8 +1135,8 @@ static int kvm_gmem_folio_prepare_attributes(struct inode *inode, pgoff_t start,
 	return 0;
 }
 
-static int kvm_gmem_ram_pfn_prepare_attributes(struct inode *inode, pgoff_t start,
-		size_t nr_pages, u64 attrs, pgoff_t *err_index)
+static int kvm_gmem_pfn_prepare_attributes(struct inode *inode, pgoff_t start,
+					   size_t nr_pages, u64 attrs, pgoff_t *err_index)
 {
 	if (!(attrs & KVM_MEMORY_ATTRIBUTE_PRIVATE))
 		return 0;
@@ -1170,15 +1172,24 @@ static const struct kvm_gmem_backing_ops kvm_gmem_folio_backing_ops = {
 static const struct kvm_gmem_backing_ops kvm_gmem_ram_pfn_backing_ops = {
 	.populate = kvm_gmem_ram_pfn_populate,
 	.revoke = kvm_gmem_ram_pfn_revoke,
-	.prepare_attributes = kvm_gmem_ram_pfn_prepare_attributes,
+	.prepare_attributes = kvm_gmem_pfn_prepare_attributes,
 	.make_private = kvm_gmem_noop_conversion,
 	.make_shared = kvm_gmem_ram_pfn_make_shared,
 	.prepare_private_pfn = kvm_gmem_ram_prepare_private_pfn,
 };
 
+static const struct kvm_gmem_backing_ops kvm_gmem_device_backing_ops = {
+	.prepare_attributes = kvm_gmem_pfn_prepare_attributes,
+	.make_private = kvm_gmem_device_make_private,
+	.make_shared = kvm_gmem_noop_conversion,
+};
+
 static const struct kvm_gmem_backing_ops *
 kvm_gmem_select_backing_ops(struct gmem_inode *gi)
 {
+	if (gmem_provider_is_device(gi))
+		return &kvm_gmem_device_backing_ops;
+
 	if (kvm_gmem_is_pfn_mode(gi))
 		return &kvm_gmem_ram_pfn_backing_ops;
 
@@ -1484,6 +1495,72 @@ void __weak kvm_arch_gmem_device_bind(struct kvm_memory_slot *slot,
 {
 }
 
+int __weak kvm_arch_gmem_device_map(struct kvm *kvm,
+				    const struct kvm_memory_slot *slot,
+				    u64 gpa, u64 size, u64 pa)
+{
+	return -EOPNOTSUPP;
+}
+
+/* The inode lock protects attributes and the offset-to-GPA bindings. */
+static int kvm_gmem_device_make_private(struct inode *inode, pgoff_t start,
+					pgoff_t end)
+{
+	struct gmem_inode *gi = GMEM_I(inode);
+	struct gmem_file *f;
+	void *entry;
+
+	MA_STATE(mas, &gi->attributes, start, start);
+
+	mas_for_each(&mas, entry, end - 1) {
+		pgoff_t first = max(start, mas.index);
+		pgoff_t last = min(end, mas.last + 1);
+
+		if (kvm_gmem_get_attributes(inode, entry) & KVM_MEMORY_ATTRIBUTE_PRIVATE)
+			continue;
+
+		while (first < last) {
+			struct kvm_memory_slot *slot = NULL;
+			kvm_pfn_t pfn, next_pfn;
+			pgoff_t high, next;
+			u64 gpa;
+			int ret;
+
+			kvm_gmem_for_each_file(f, inode) {
+				slot = xa_load(&f->bindings, first);
+				if (slot)
+					break;
+			}
+			if (!slot)
+				return -EINVAL;
+
+			high = min(last, slot->gmem.pgoff + slot->npages);
+			gpa = (slot->base_gfn + first - slot->gmem.pgoff) << PAGE_SHIFT;
+			ret = gmem_provider_get_pfn(gi, first, &pfn, NULL);
+			if (ret)
+				return ret;
+
+			/* Map a contiguous provider PFN run within this binding. */
+			for (next = first + 1; next < high; next++) {
+				ret = gmem_provider_get_pfn(gi, next, &next_pfn, NULL);
+				if (ret)
+					return ret;
+				if (next_pfn != pfn + next - first)
+					break;
+			}
+			high = next;
+			ret = kvm_arch_gmem_device_map(gi->device.kvm, slot, gpa,
+						       (u64)(high - first) << PAGE_SHIFT,
+						       PFN_PHYS(pfn));
+			if (ret)
+				return ret;
+
+			first = high;
+		}
+	}
+	return 0;
+}
+
 static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags, int resource_fd)
 {
 	static const char *name = "[kvm-gmem]";
-- 
2.43.0

[RFC PATCH v3 08/13] KVM: guest_memfd: Track private device ranges after attribute updates

[Aneesh Kumar K.V (Arm)]

The following device access callbacks need to report whether any private
ranges remain without acquiring the inode invalidate lock under a
provider lock.

Add device_has_private as a cached summary of the committed attributes.
Add an optional update_attributes() backing operation and call it after
storing new attributes on successful conversion. Device backing scans
the attribute tree to refresh the summary; folio and RAM PFN backing
leave the operation unset.

Run the hook after both private and shared conversions, before releasing
conversion locks. The scan must see the newly stored attributes, including
the removal of the last private range. Keep this bookkeeping separate
from the architecture mapping operations.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 virt/kvm/guest_memfd.c | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 3bf26490f4e9..af1e4798ace1 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -44,6 +44,8 @@ struct kvm_gmem_backing_ops {
 	/* Prepare private RAM state after lookup, before returning a guest PFN. */
 	int (*prepare_private_pfn)(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
 				   kvm_pfn_t nr_pages);
+	/* Called after the common code stores the new attributes. */
+	void (*update_attributes)(struct inode *inode);
 };
 
 struct gmem_inode {
@@ -56,6 +58,7 @@ struct gmem_inode {
 	const struct kvm_gmem_backing_ops *backing_ops;
 
 	struct guest_memfd_device device;
+	bool device_has_private;
 
 	u64 flags;
 	/*
@@ -80,6 +83,7 @@ static bool gmem_provider_is_device(struct gmem_inode *gi)
 	return gi->provider_ops && gi->provider_ops->device_memory;
 }
 
+static void kvm_gmem_device_update_attributes(struct inode *inode);
 static int kvm_gmem_device_make_private(struct inode *inode, pgoff_t start, pgoff_t end);
 
 struct guest_memfd_device *kvm_gmem_get_device(struct inode *inode)
@@ -1182,6 +1186,7 @@ static const struct kvm_gmem_backing_ops kvm_gmem_device_backing_ops = {
 	.prepare_attributes = kvm_gmem_pfn_prepare_attributes,
 	.make_private = kvm_gmem_device_make_private,
 	.make_shared = kvm_gmem_noop_conversion,
+	.update_attributes = kvm_gmem_device_update_attributes,
 };
 
 static const struct kvm_gmem_backing_ops *
@@ -1241,8 +1246,11 @@ static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 	kvm_gmem_invalidate_start(inode, start, end, filter);
 
 	r = kvm_gmem_convert_attributes(inode, start, end, attrs);
-	if (!r)
+	if (!r) {
 		mas_store_prealloc(&mas, xa_mk_value(attrs));
+		if (gi->backing_ops->update_attributes)
+			gi->backing_ops->update_attributes(inode);
+	}
 
 	if (r)
 		mas_destroy(&mas);
@@ -1502,6 +1510,24 @@ int __weak kvm_arch_gmem_device_map(struct kvm *kvm,
 	return -EOPNOTSUPP;
 }
 
+static void kvm_gmem_device_update_attributes(struct inode *inode)
+{
+	struct gmem_inode *gi = GMEM_I(inode);
+	void *entry;
+	bool private = false;
+
+	MA_STATE(mas, &gi->attributes, 0, 0);
+
+	mas_for_each(&mas, entry, ULONG_MAX) {
+		if (kvm_gmem_get_attributes(&gi->vfs_inode, entry) &
+		    KVM_MEMORY_ATTRIBUTE_PRIVATE) {
+			private = true;
+			break;
+		}
+	}
+	WRITE_ONCE(gi->device_has_private, private);
+}
+
 /* The inode lock protects attributes and the offset-to-GPA bindings. */
 static int kvm_gmem_device_make_private(struct inode *inode, pgoff_t start,
 					pgoff_t end)
@@ -2042,6 +2068,7 @@ static struct inode *kvm_gmem_alloc_inode(struct super_block *sb)
 	gi->provider_ops = NULL;
 	gi->backing_ops = NULL;
 	memset(&gi->device, 0, sizeof(gi->device));
+	gi->device_has_private = false;
 	INIT_LIST_HEAD(&gi->gmem_file_list);
 	return &gi->vfs_inode;
 }
-- 
2.43.0

[RFC PATCH v3 09/13] KVM: guest_memfd: Expose device provider private state

[Aneesh Kumar K.V (Arm)]

Expose a callback to query whether private ranges remain. Initialize it
before attaching the device provider so the vDEVICE can coordinate
protected cleanup. Keep the summary inode-owned and valid until release;
providers may access the VM only during attach and core callbacks.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 include/linux/guest_memfd.h |  7 +++++++
 virt/kvm/guest_memfd.c      | 10 ++++++++++
 2 files changed, 17 insertions(+)

diff --git a/include/linux/guest_memfd.h b/include/linux/guest_memfd.h
index 8a93004aa9e7..5e5725398ab0 100644
--- a/include/linux/guest_memfd.h
+++ b/include/linux/guest_memfd.h
@@ -24,12 +24,19 @@ struct guest_memfd_device_binding {
  * @flags: GUEST_MEMFD_FLAG_* creation flags
  * @binding: Mapping owner and device ID set by attach(), with the underlying
  *           resources pinned until release()
+ * @has_private: Check whether any committed private ranges remain
+ *
+ * The VM is valid during attach() and core-initiated callbacks. Providers
+ * must not use this pointer for asynchronous VM access.
+ * release() may run after the VM reference has been dropped. Inode-owned
+ * metadata, including the private-range summary, remains valid until then.
  */
 struct guest_memfd_device {
 	struct kvm *kvm;
 	u64 size;
 	u64 flags;
 	struct guest_memfd_device_binding binding;
+	bool (*has_private)(struct guest_memfd_device *gmem);
 };
 
 /**
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index af1e4798ace1..1322576350b8 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -85,6 +85,7 @@ static bool gmem_provider_is_device(struct gmem_inode *gi)
 
 static void kvm_gmem_device_update_attributes(struct inode *inode);
 static int kvm_gmem_device_make_private(struct inode *inode, pgoff_t start, pgoff_t end);
+static bool gmem_device_has_private(struct guest_memfd_device *gdev);
 
 struct guest_memfd_device *kvm_gmem_get_device(struct inode *inode)
 {
@@ -176,6 +177,7 @@ static int kvm_gmem_attach_resource(struct kvm *kvm, struct file *file,
 			gi->device.kvm = kvm;
 			gi->device.size = i_size_read(inode);
 			gi->device.flags = gi->flags;
+			gi->device.has_private = gmem_device_has_private;
 		}
 		provider = ops->attach(inode, resource_file);
 	}
@@ -1510,6 +1512,14 @@ int __weak kvm_arch_gmem_device_map(struct kvm *kvm,
 	return -EOPNOTSUPP;
 }
 
+/* The attribute tree owns mapping state; this summary is read under provider locks. */
+static bool gmem_device_has_private(struct guest_memfd_device *gdev)
+{
+	struct gmem_inode *gi = container_of(gdev, struct gmem_inode, device);
+
+	return READ_ONCE(gi->device_has_private);
+}
+
 static void kvm_gmem_device_update_attributes(struct inode *inode)
 {
 	struct gmem_inode *gi = GMEM_I(inode);
-- 
2.43.0

[RFC PATCH v3 10/13] KVM: guest_memfd: Support range revocation of device mappings

[Aneesh Kumar K.V (Arm)]

Give device backing a revoke operation that removes host and both shared
and private guest translations under provider exclusion, without reclaiming
MMIO PFNs or changing attributes. Keep fallocate and hole punching
unsupported for device memory.

Range revocation does not exclude new faults after returning. Providers
must preserve the underlying resource and call without provider locks held,
since inode exclusion precedes provider exclusion. Private attributes
continue to describe ownership until explicitly changed or the file closes.

Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 virt/kvm/guest_memfd.c | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)

diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 1322576350b8..cfb7f34c0180 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -559,6 +559,18 @@ static int kvm_gmem_ram_pfn_revoke(struct inode *inode, pgoff_t start, pgoff_t e
 	return 0;
 }
 
+static int kvm_gmem_device_revoke(struct inode *inode, pgoff_t start, pgoff_t end,
+		const unsigned long *pfn, bool put_pfn_range)
+{
+	/*
+	 * Common invalidation handles shared and private guest mappings. Only
+	 * host userspace mappings need to be unmapped here; MMIO has no RAM
+	 * pages to reclaim or provider allocations to release.
+	 */
+	unmap_mapping_pages(inode->i_mapping, start, end - start, true);
+	return 0;
+}
+
 static int __kvm_gmem_revoke(struct inode *inode, pgoff_t start,
 			      size_t nr_pages, const unsigned long *pfn,
 			      bool put_pfn_range)
@@ -622,7 +634,11 @@ static long kvm_gmem_punch_hole(struct inode *inode, loff_t offset, loff_t len)
  *
  * Must be called before accessing memory contents. Keeping the mapping from
  * @start to @pfn constant is the responsibility of the caller.
- * .put_pfn_range() will not be called.
+ * .put_pfn_range() will not be called. For device memory, this revokes host
+ * and both shared and private guest mappings under provider exclusion, without
+ * changing attributes or releasing BAR PFNs. Call without provider locks held:
+ * inode exclusion precedes provider exclusion. This does not exclude new faults
+ * after returning; the provider must preserve the backing resource.
  *
  * Return: 0 on success, or a negative error code on failure.
  */
@@ -1185,6 +1201,7 @@ static const struct kvm_gmem_backing_ops kvm_gmem_ram_pfn_backing_ops = {
 };
 
 static const struct kvm_gmem_backing_ops kvm_gmem_device_backing_ops = {
+	.revoke = kvm_gmem_device_revoke,
 	.prepare_attributes = kvm_gmem_pfn_prepare_attributes,
 	.make_private = kvm_gmem_device_make_private,
 	.make_shared = kvm_gmem_noop_conversion,
-- 
2.43.0

[RFC PATCH v3 11/13] KVM: guest_memfd: Add shared device mmap through PFN providers

[Aneesh Kumar K.V (Arm)]

Map shared device memory through the existing get_pfn callback. Set
device VMA flags and protections in the core, and retain inode exclusion
through PFN lookup and PTE insertion. Validate BAR access at fault time;
memslot creation validates the complete interval separately. Provider
errors use the same fault handling as RAM backing. An unavailable BAR
returns ENXIO, which becomes SIGBUS rather than an unproductive fault
retry.

Private pages remain inaccessible to userspace. Conversion revokes host
mappings before changing attributes, and VMA references retain the file
and provider lifetime. VFIO fd mmap remains excluded.

Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 include/linux/guest_memfd.h |  9 +++++++--
 virt/kvm/guest_memfd.c      | 19 ++++++++++++++++---
 2 files changed, 23 insertions(+), 5 deletions(-)

diff --git a/include/linux/guest_memfd.h b/include/linux/guest_memfd.h
index 5e5725398ab0..5e961f163d2a 100644
--- a/include/linux/guest_memfd.h
+++ b/include/linux/guest_memfd.h
@@ -2,6 +2,7 @@
 #ifndef _LINUX_GUEST_MEMFD_H
 #define _LINUX_GUEST_MEMFD_H
 
+#include <linux/mm_types.h>
 #include <linux/types.h>
 
 struct file;
@@ -47,10 +48,14 @@ struct guest_memfd_device {
  *          through kvm_gmem_get_device().
  * @release: Release provider context and unpin any resources. Must not access
  *           the VM here; its file reference may be gone.
- * @device_memory: Backing is device MMIO rather than RAM.
+ * @device_memory: Backing is device MMIO rather than RAM. The core maps its
+ *                 userspace VMAs noncached and decrypted, with device flags.
+ *                 Shared userspace faults are resolved through @get_pfn, with
+ *                 the same error handling as RAM providers.
  * @prepare_memory_region: Optional check of a memory range and preparation of
  *                         access to its backing memory. Called after common
- *                         file checks and before publishing the memslot.
+ *                         file checks and before publishing the memslot,
+ *                         with the inode invalidate lock held for read.
  *                         Offset and size are in bytes. The flags contain only
  *                         KVM_MEM_LOG_DIRTY_PAGES and KVM_MEM_READONLY bits.
  *                         Return 0 on success or a negative error code.
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index cfb7f34c0180..7989c093a209 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -7,6 +7,7 @@
 #include <linux/kvm_host.h>
 #include <linux/maple_tree.h>
 #include <linux/mempolicy.h>
+#include <linux/mutex.h>
 #include <linux/pseudo_fs.h>
 #include <linux/pagemap.h>
 #include <linux/swap.h>
@@ -930,12 +931,16 @@ static const struct vm_operations_struct kvm_gmem_vm_ops = {
 #endif
 };
 
+static const struct vm_operations_struct kvm_gmem_device_vm_ops = {
+	.fault = kvm_gmem_fault_user_mapping,
+};
+
 static int kvm_gmem_mmap(struct file *file, struct vm_area_struct *vma)
 {
-	struct gmem_inode *gi = GMEM_I(file_inode(file));
+	struct inode *inode = file_inode(file);
+	struct gmem_inode *gi = GMEM_I(inode);
 
-	if (gmem_provider_is_device(gi) ||
-	    !kvm_gmem_supports_mmap(file_inode(file)))
+	if (!kvm_gmem_supports_mmap(inode))
 		return -ENODEV;
 
 	if ((vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) !=
@@ -946,6 +951,14 @@ static int kvm_gmem_mmap(struct file *file, struct vm_area_struct *vma)
 	if (kvm_gmem_is_pfn_mode(gi))
 		vm_flags_set(vma, VM_PFNMAP);
 
+	if (gmem_provider_is_device(gi)) {
+		vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
+		vma->vm_page_prot = pgprot_decrypted(vma->vm_page_prot);
+		vm_flags_set(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP);
+		vma->vm_ops = &kvm_gmem_device_vm_ops;
+		return 0;
+	}
+
 	vma->vm_ops = &kvm_gmem_vm_ops;
 
 	return 0;
-- 
2.43.0

[RFC PATCH v3 12/13] KVM: guest_memfd: Remove device mappings on final close

[Aneesh Kumar K.V (Arm)]

Remove device mappings before dropping the file's VM reference. Keep
memslot bindings available so invalidation can resolve device ranges to
guest addresses.

Hold the inode invalidate lock and provider exclusion across one common
invalidation interval covering both shared and private mappings. Clear
the slot file pointers before invalidation and remove bindings afterward.

Add an optional backing release() operation after invalidation. Device
backing clears its inode-owned private-range summary there, allowing TDI
cleanup after the last file user is gone. Do not convert file attributes
or reclaim BAR PFNs. Provider resources remain pinned until inode cleanup.
Guard callbacks for failed attachment before provider setup completes.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 virt/kvm/guest_memfd.c | 22 +++++++++++++++++++---
 1 file changed, 19 insertions(+), 3 deletions(-)

diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 7989c093a209..22793d3c6aff 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -47,6 +47,8 @@ struct kvm_gmem_backing_ops {
 				   kvm_pfn_t nr_pages);
 	/* Called after the common code stores the new attributes. */
 	void (*update_attributes)(struct inode *inode);
+	/* Finish file cleanup after invalidation, under provider exclusion. */
+	void (*release)(struct inode *inode);
 };
 
 struct gmem_inode {
@@ -86,6 +88,7 @@ static bool gmem_provider_is_device(struct gmem_inode *gi)
 
 static void kvm_gmem_device_update_attributes(struct inode *inode);
 static int kvm_gmem_device_make_private(struct inode *inode, pgoff_t start, pgoff_t end);
+static void kvm_gmem_device_release(struct inode *inode);
 static bool gmem_device_has_private(struct guest_memfd_device *gdev);
 
 struct guest_memfd_device *kvm_gmem_get_device(struct inode *inode)
@@ -772,9 +775,11 @@ static long kvm_gmem_fallocate(struct file *file, int mode, loff_t offset,
 static int kvm_gmem_release(struct inode *inode, struct file *file)
 {
 	struct gmem_file *f = file->private_data;
+	struct gmem_inode *gi = GMEM_I(inode);
 	struct kvm_memory_slot *slot;
 	struct kvm *kvm = f->kvm;
 	unsigned long index;
+	enum kvm_gfn_range_filter filter = kvm_gmem_get_all_gfns_filter(inode);
 
 	/*
 	 * Prevent concurrent attempts to *unbind* a memslot.  This is the last
@@ -793,17 +798,21 @@ static int kvm_gmem_release(struct inode *inode, struct file *file)
 
 	filemap_invalidate_lock(inode->i_mapping);
 
+	gmem_provider_lock(gi);
+
 	xa_for_each(&f->bindings, index, slot)
 		WRITE_ONCE(slot->gmem.file, NULL);
 
 	/*
-	 * All in-flight operations are gone and new bindings can be created.
+	 * All in-flight operations are gone and no new bindings can be created.
 	 * Zap all SPTEs pointed at by this file.  Do not free the backing
 	 * memory, as its lifetime is associated with the inode, not the file.
 	 */
-	__kvm_gmem_invalidate_start(f, 0, -1ul,
-				    kvm_gmem_get_all_gfns_filter(inode));
+	__kvm_gmem_invalidate_start(f, 0, -1ul, filter);
 	__kvm_gmem_invalidate_end(f, 0, -1ul);
+	if (gi->backing_ops && gi->backing_ops->release)
+		gi->backing_ops->release(inode);
+	gmem_provider_unlock(gi);
 
 	list_del(&f->entry);
 
@@ -1219,6 +1228,7 @@ static const struct kvm_gmem_backing_ops kvm_gmem_device_backing_ops = {
 	.make_private = kvm_gmem_device_make_private,
 	.make_shared = kvm_gmem_noop_conversion,
 	.update_attributes = kvm_gmem_device_update_attributes,
+	.release = kvm_gmem_device_release,
 };
 
 static const struct kvm_gmem_backing_ops *
@@ -1627,6 +1637,12 @@ static int kvm_gmem_device_make_private(struct inode *inode, pgoff_t start,
 	return 0;
 }
 
+/* No file users remain, so private attributes cannot create new mappings. */
+static void kvm_gmem_device_release(struct inode *inode)
+{
+	WRITE_ONCE(GMEM_I(inode)->device_has_private, false);
+}
+
 static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags, int resource_fd)
 {
 	static const char *name = "[kvm-gmem]";
-- 
2.43.0

[RFC PATCH v3 13/13] KVM: guest_memfd: Invalidate guest_memfd mappings before removing memslot bindings

[Aneesh Kumar K.V (Arm)]

Unmap guest_memfd ranges before removing their offset-to-GPA bindings.
Invalidate shared and private mappings while holding inode and provider
exclusion, using the same common invalidation path as revocation.

Do not convert device attributes when deleting a memslot. Private file
attributes continue to block TDI UNLOCK after unbind; userspace must
explicitly share those ranges or release the last file reference.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 virt/kvm/guest_memfd.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 22793d3c6aff..e31234212dcf 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -1860,6 +1860,8 @@ void kvm_gmem_unbind(struct kvm_memory_slot *slot)
 	unsigned long start = slot->gmem.pgoff;
 	unsigned long end = start + slot->npages;
 	struct gmem_file *f;
+	struct gmem_inode *gi;
+	enum kvm_gfn_range_filter filter;
 
 	/*
 	 * Nothing to do if the underlying file was _already_ closed, as
@@ -1879,8 +1881,13 @@ void kvm_gmem_unbind(struct kvm_memory_slot *slot)
 	 * drops slots_lock, so there's no need to verify the file is live.
 	 */
 	f = file->private_data;
+	gi = GMEM_I(file_inode(file));
+	filter = kvm_gmem_get_all_gfns_filter(file_inode(file));
 
 	filemap_invalidate_lock(file->f_mapping);
+	gmem_provider_lock(gi);
+	__kvm_gmem_invalidate_start(f, start, end, filter);
+	__kvm_gmem_invalidate_end(f, start, end);
 	xa_store_range(&f->bindings, start, end - 1, NULL, GFP_KERNEL);
 
 	/*
@@ -1888,6 +1895,7 @@ void kvm_gmem_unbind(struct kvm_memory_slot *slot)
 	 * cannot see this memslot.
 	 */
 	WRITE_ONCE(slot->gmem.file, NULL);
+	gmem_provider_unlock(gi);
 	filemap_invalidate_unlock(file->f_mapping);
 }
 
-- 
2.43.0