[RFC PATCH v2 00/13] KVM/VFIO: guest_memfd support for device MMIO resources

[RFC PATCH v2 00/13] KVM/VFIO: guest_memfd support for device MMIO resources

[Aneesh Kumar K.V (Arm)]

This series adds a guest_memfd device-resource framework for exposing PCI
BAR memory to guests. The intended use is confidential device assignment,
where KVM needs to coordinate private device mappings with shared access
through VFIO. The Arm CCA backend is provided separately; these patches
contain the guest_memfd, VFIO and IOMMUFD interfaces it uses.

The guest_memfd framework and Arm CCA backend are available together at:

  https://git.gitlab.arm.com/linux-arm/linux-cca.git cca/topics/cca-da-gmem-backend

Userspace creates a device-backed guest_memfd using the ordinary VFIO
device fd as resource_fd, with GUEST_MEMFD_FLAG_USE_RESOURCE and
GUEST_MEMFD_FLAG_INIT_SHARED. The resource mapping superblock selects its
provider; get_pfn identifies PFN-backed memory. Eligible BAR intervals are
registered as guest_memfd memslots. File offsets use the existing VFIO
region-offset namespace, allowing one guest_memfd to cover multiple BAR
intervals. The device-backed guest_memfd fd rejects mmap and fallocate;
shared guest faults obtain bare BAR PFNs from the provider instead of
resolving the memslot HVA or allocating folios.

IOMMUFD resolves and pins the device's existing vDEVICE after its backend
validates the VM association. Its MMIO mutex coordinates conversion and
device state changes. VFIO supplies BAR PFNs, validates eligible ranges
and excludes dma-buf exports while a guest_memfd provider is attached.

Without GUEST_MEMFD_FLAG_MMAP, attachment rejects existing VFIO BAR VMAs,
and new VFIO mmap calls. With the flag, existing and new BAR VMAs are
allowed through the VFIO fd. Attachment invalidates existing BAR mappings,
including huge mappings. Subsequent faults use base-page mappings so each
page's shared-range attributes are checked separately. Mapping the
PFN-backed guest_memfd fd remains disabled (We can decide on this later).

VFIO BAR read/write, ioeventfd and userspace mapping faults check the
requested range's guest_memfd attributes.

This series depends on:
* Ackerley Tng's "Allow guest_memfd to be created using a resource (pool)
  fd" RFC, which provides the tmpfs resource-provider API:
  https://lore.kernel.org/all/20260925-gmem-tmpfs-backend-v1-0-d36159822d18@google.com/
* The IOMMUFD/TSM series, which provides the vIOMMU and vDEVICE interfaces:
  https://lore.kernel.org/all/20261008055955.4014342-1-aneesh.kumar@kernel.org/

Notes:
* Codex assisted with implementation, review and series organization.

* Changes from v1:
https://lore.kernel.org/all/20261010072504.536230-1-aneesh.kumar@kernel.org
* Require GUEST_MEMFD_FLAG_MMAP to allow VFIO BAR mmap while a provider is
  attached; disable it by default.
* Use the common guest_memfd_provider_operations for folio and PFN
  providers. Select operations through the resource mapping superblock
  and forward VFIO operations to the device driver.

Cc: Ackerley Tng <ackerleytng@google.com>
Cc: Alex Williamson <alex@shazbot.org>
Cc: David Woodhouse <dwmw2@infradead.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: "Joerg Roedel (AMD)" <joro@8bytes.org>
Cc: Kevin Tian <kevin.tian@intel.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Robin Murphy <robin.murphy@arm.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Will Deacon <will@kernel.org>
Cc: Alexey Kardashevskiy <aik@amd.com>
Cc: Xu Yilun <yilun.xu@linux.intel.com>
Cc: Catalin Marinas <catalin.marinas@arm.com> 
Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
Cc: Steven Price <steven.price@arm.com>
Cc: Fred Griffoul <griffoul@gmail.com> 
Cc: iommu@lists.linux.dev
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev

Aneesh Kumar K.V (Arm) (13):
  KVM: guest_memfd: Support PFN-backed resource providers
  KVM: guest_memfd: Support private/shared conversion of device memory
  iommufd: Support MMIO provider attachment to vDEVICEs
  KVM: guest_memfd: Add device provider access callbacks
  vfio/pci: Track userspace BAR mappings
  vfio: Dispatch guest_memfd provider operations through the filesystem
  vfio/pci: Restrict host BAR access for guest_memfd providers
  vfio/pci: Provide guest_memfd backing for PCI BARs
  KVM: guest_memfd: Add a provider detach callback
  vfio/pci: Detach guest_memfd access before teardown
  KVM: guest_memfd: Remove device mappings on final close
  KVM: Invalidate guest_memfd mappings before removing memslot bindings
  vfio/pci: Allow VFIO BAR mmap with an attached guest_memfd

 Documentation/virt/kvm/api.rst          |  26 +-
 drivers/iommu/iommufd/viommu.c          |  86 ++++
 drivers/vfio/pci/Kconfig                |  11 +
 drivers/vfio/pci/Makefile               |   2 +
 drivers/vfio/pci/vfio_pci.c             |   3 +
 drivers/vfio/pci/vfio_pci_core.c        |  34 ++
 drivers/vfio/pci/vfio_pci_dmabuf.c      |  10 +
 drivers/vfio/pci/vfio_pci_gmem.c        | 362 +++++++++++++++
 drivers/vfio/pci/vfio_pci_gmem.h        |  26 ++
 drivers/vfio/pci/vfio_pci_gmem_access.c | 135 ++++++
 drivers/vfio/pci/vfio_pci_rdwr.c        |  13 +-
 drivers/vfio/vfio_main.c                | 121 ++++-
 include/linux/guest_memfd.h             |  60 ++-
 include/linux/iommufd.h                 |  26 ++
 include/linux/kvm_host.h                |  11 +
 include/linux/vfio.h                    |   4 +
 include/linux/vfio_pci_core.h           |  30 ++
 mm/shmem.c                              |   3 +-
 virt/kvm/guest_memfd.c                  | 566 ++++++++++++++++++++++--
 virt/kvm/guest_memfd.h                  |   6 +
 virt/kvm/kvm_main.c                     |   4 +
 21 files changed, 1492 insertions(+), 47 deletions(-)
 create mode 100644 drivers/vfio/pci/vfio_pci_gmem.c
 create mode 100644 drivers/vfio/pci/vfio_pci_gmem.h
 create mode 100644 drivers/vfio/pci/vfio_pci_gmem_access.c

-- 
2.43.0

[RFC PATCH v2 01/13] KVM: guest_memfd: Support PFN-backed resource providers

[Aneesh Kumar K.V (Arm)]

Device resources such as PCI BARs supply PFNs directly and cannot use
the folio allocation path. Extend guest_memfd provider operations with
bind() and get_pfn() callbacks and identify the backing type from the
callbacks supplied by the provider.

Require providers to supply either folio allocation or PFN lookup,
along with the callbacks appropriate to that backing type. PFN providers
require INIT_SHARED and architecture support for protected device
mappings.

Pass creation flags to providers so they can control mappings through
the resource fd. Keep mmap on PFN-backed guest_memfd files disabled.

Resolve shared faults through get_pfn() without taking folio references.
Reject dirty logging and read-only device memslots.

Provider release may run after file release and must not access the VM.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 Documentation/virt/kvm/api.rst |   4 +
 include/linux/guest_memfd.h    |  39 ++++++--
 include/linux/kvm_host.h       |   6 ++
 mm/shmem.c                     |   3 +-
 virt/kvm/guest_memfd.c         | 165 +++++++++++++++++++++++++++------
 5 files changed, 180 insertions(+), 37 deletions(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index 5ece28765d2c..078f672a115b 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -6529,6 +6529,10 @@ specified via KVM_CREATE_GUEST_MEMFD.  Currently defined flags:
                                 require noswap and huge=never mount options.
   ============================= ================================================
 
+With USE_RESOURCE, the resource mapping superblock supplies provider
+operations. Callbacks identify folio or PFN backing. PFN providers require
+INIT_SHARED and a VM supporting protected resource mappings; mmap is disabled.
+
 When the KVM MMU performs a PFN lookup to service a guest fault, the fault will
 always be consumed from guest_memfd, regardless of whether it is a shared or a
 private fault (unless in-place conversion is disabled and the backing
diff --git a/include/linux/guest_memfd.h b/include/linux/guest_memfd.h
index 60eb4f008c24..1bc9a23afa4e 100644
--- a/include/linux/guest_memfd.h
+++ b/include/linux/guest_memfd.h
@@ -7,23 +7,46 @@
 struct file;
 struct folio;
 struct mempolicy;
+struct kvm;
+
+/* Binding data stays valid only while the provider is active. */
+struct guest_memfd_device_binding {
+	/* Architecture-specific mapping owner, e.g. an RMM vDEVICE handle. */
+	unsigned long owner;
+	u64 vdev_id;
+};
+
+struct guest_memfd {
+	struct kvm *kvm;
+	u64 size;
+	u64 flags;
+};
 
 /**
- * struct guest_memfd_provider_operations - Operations for external memory providers
- * @attach: Attach to a resource file, perform filesystem-specific validation,
- *          and return an opaque provider context.
- * @release: Release provider context and unpin any resources.
- * @alloc_folio: Allocate an uninserted folio for a given index and NUMA policy.
- * @invalidate_folio: Notify provider that a folio has been invalidated.
+ * struct guest_memfd_provider_operations - External memory provider operations
+ * @attach: Validate the resource and attach it to the core context. Return
+ *          an opaque provider or ERR_PTR(). Success pins backing resources
+ *          and these operations until release().
+ * @release: Release the provider context and backing references. Must not
+ *           access the VM or invoke callbacks that do; the file may be gone.
+ * @alloc_folio: Allocate an uninserted folio for an index and NUMA policy.
+ * @invalidate_folio: Notify the folio provider before releasing a folio.
+ * @bind: Validate a PFN interval and return its protected mapping owner.
+ * @get_pfn: Resolve a shared resource offset without taking a folio reference.
  *
- * Used by filesystems and device drivers that provide memory for guest_memfd.
+ * Provider operations supply exactly one of alloc_folio() and get_pfn().
+ * PFN providers start shared and require the bind and conversion callbacks
+ * supported by the core. Folio providers use core allocation and conversion.
  */
 struct guest_memfd_provider_operations {
-	void *(*attach)(struct file *resource_file);
+	void *(*attach)(struct file *resource, struct guest_memfd *gmem);
 	void (*release)(void *provider);
 	struct folio *(*alloc_folio)(void *provider, pgoff_t index,
 				     struct mempolicy *mpol);
 	void (*invalidate_folio)(void *provider, struct folio *folio);
+	int (*bind)(void *provider, u64 offset, u64 size,
+		struct guest_memfd_device_binding *binding);
+	int (*get_pfn)(void *provider, u64 offset, unsigned long *pfn);
 };
 
 #endif /* _LINUX_GUEST_MEMFD_H */
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 1dda54e8dee8..2aa728de75e8 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -56,6 +56,7 @@
  */
 #define KVM_MEMSLOT_INVALID			(1UL << 16)
 #define KVM_MEMSLOT_GMEM_ONLY			(1UL << 17)
+#define KVM_MEMSLOT_GMEM_DEVICE			(1UL << 18)
 
 /*
  * Bit 63 of the memslot generation number is an "update in-progress flag",
@@ -2605,9 +2606,14 @@ static inline bool kvm_is_private_gfn(struct kvm *kvm, gfn_t gfn)
 #endif /* kvm_arch_has_private_mem */
 
 #ifdef CONFIG_KVM_GUEST_MEMFD
+struct guest_memfd_device_binding;
+
 bool kvm_gmem_is_private_gfn(struct kvm *kvm, gfn_t gfn);
 bool kvm_gmem_range_has_attributes(struct kvm_memory_slot *slot, gfn_t start,
 				   gfn_t end, u64 attributes);
+bool kvm_arch_gmem_device_supported(struct kvm *kvm);
+void kvm_arch_gmem_device_bind(struct kvm_memory_slot *slot,
+		const struct guest_memfd_device_binding *binding);
 int kvm_gmem_set_attributes(struct kvm_memory_slot *slot, gfn_t start,
 			    gfn_t end, u64 attributes);
 bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm);
diff --git a/mm/shmem.c b/mm/shmem.c
index 493b72af52aa..d8f8a3b1fe59 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -5239,7 +5239,8 @@ static const struct inode_operations shmem_special_inode_operations = {
 };
 
 #ifdef CONFIG_KVM_GUEST_MEMFD
-static void *shmem_gmem_attach(struct file *resource_file)
+static void *shmem_gmem_attach(struct file *resource_file,
+		struct guest_memfd *gmem)
 {
 	struct inode *inode = file_inode(resource_file);
 	struct shmem_sb_info *sbinfo;
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index a65a1eda8b90..9a8b66bd623a 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -2,6 +2,7 @@
 #include <linux/anon_inodes.h>
 #include <linux/backing-dev.h>
 #include <linux/falloc.h>
+#include <linux/export.h>
 #include <linux/fs.h>
 #include <linux/guest_memfd.h>
 #include <linux/kvm_host.h>
@@ -38,6 +39,7 @@ struct gmem_inode {
 	struct inode vfs_inode;
 	struct list_head gmem_file_list;
 
+	struct guest_memfd device;
 	void *provider;
 	const struct guest_memfd_provider_operations *provider_ops;
 
@@ -56,6 +58,11 @@ static __always_inline struct gmem_inode *GMEM_I(struct inode *inode)
 	return container_of(inode, struct gmem_inode, vfs_inode);
 }
 
+static bool gmem_provider_has_pfns(struct gmem_inode *gi)
+{
+	return gi->provider_ops && gi->provider_ops->get_pfn;
+}
+
 static inline struct folio *gmem_provider_alloc_folio(struct gmem_inode *gi,
 						      pgoff_t index,
 						      struct mempolicy *mpol)
@@ -75,7 +82,7 @@ static inline void gmem_provider_invalidate_folio(struct gmem_inode *gi,
 
 static inline void gmem_provider_release(struct gmem_inode *gi)
 {
-	if (gi->provider_ops && gi->provider_ops->release)
+	if (gi->provider_ops)
 		gi->provider_ops->release(gi->provider);
 }
 
@@ -189,6 +196,9 @@ static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index)
 	struct mempolicy *policy;
 	struct folio *folio;
 
+	if (gmem_provider_has_pfns(gi))
+		return ERR_PTR(-EOPNOTSUPP);
+
 	/*
 	 * Fast-path: See if folio is already present in mapping to avoid
 	 * policy_lookup.
@@ -237,6 +247,8 @@ static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index)
 
 static enum kvm_gfn_range_filter kvm_gmem_get_all_gfns_filter(struct inode *inode)
 {
+	if (gmem_provider_has_pfns(GMEM_I(inode)))
+		return KVM_FILTER_SHARED;
 	if (gmem_in_place_conversion)
 		return KVM_FILTER_SHARED | KVM_FILTER_PRIVATE;
 
@@ -393,6 +405,9 @@ static long kvm_gmem_fallocate(struct file *file, int mode, loff_t offset,
 {
 	int ret;
 
+	if (gmem_provider_has_pfns(GMEM_I(file_inode(file))))
+		return -EOPNOTSUPP;
+
 	if (!(mode & FALLOC_FL_KEEP_SIZE))
 		return -EOPNOTSUPP;
 
@@ -477,7 +492,11 @@ DEFINE_CLASS(gmem_get_file, struct file *, if (_T) fput(_T),
 
 static bool kvm_gmem_supports_mmap(struct inode *inode)
 {
-	return GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_MMAP;
+	struct gmem_inode *gi = GMEM_I(inode);
+
+	/* PFN providers may allow resource fd mmap, but not guest_memfd mmap. */
+	return !gmem_provider_has_pfns(gi) &&
+		(gi->flags & GUEST_MEMFD_FLAG_MMAP);
 }
 
 static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf)
@@ -733,6 +752,9 @@ static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 	struct ma_state mas;
 	int r = 0;
 
+	if (gmem_provider_has_pfns(gi))
+		return -EOPNOTSUPP;
+
 	mt = &gi->attributes;
 
 	filemap_invalidate_lock(mapping);
@@ -1007,34 +1029,90 @@ static int kvm_gmem_init_inode(struct inode *inode, loff_t size, u64 flags)
 	return r;
 }
 
-static int kvm_gmem_attach_resource(struct inode *inode, int resource_fd)
+bool __weak kvm_arch_gmem_device_supported(struct kvm *kvm)
+{
+	return false;
+}
+
+void __weak kvm_arch_gmem_device_bind(struct kvm_memory_slot *slot,
+		const struct guest_memfd_device_binding *binding)
+{
+}
+
+static int kvm_gmem_validate_provider(struct kvm *kvm,
+		struct gmem_inode *gi,
+		const struct guest_memfd_provider_operations *ops)
+{
+	/* Every provider must support attachment and release. */
+	if (!ops || !ops->attach || !ops->release)
+		return -EOPNOTSUPP;
+
+	/* Providers must supply exactly one backing mechanism. */
+	if (!!ops->alloc_folio == !!ops->get_pfn)
+		return -EOPNOTSUPP;
+
+	/* Folio providers use the core's binding and conversion paths. */
+	if (!ops->get_pfn) {
+		/* PFN-specific callbacks are invalid for folio providers. */
+		if (ops->bind)
+			return -EOPNOTSUPP;
+		return 0;
+	}
+
+	/* PFN providers must support binding. */
+	if (!ops->bind)
+		return -EOPNOTSUPP;
+
+	/* PFN providers do not manage folios. */
+	if (ops->invalidate_folio)
+		return -EOPNOTSUPP;
+
+	/* The VM must support protected resource mappings. */
+	if (!kvm_arch_gmem_device_supported(kvm))
+		return -EOPNOTSUPP;
+
+	/* PFN-backed memory must start shared. */
+	if (!(gi->flags & GUEST_MEMFD_FLAG_INIT_SHARED))
+		return -EOPNOTSUPP;
+
+	return 0;
+}
+
+static int kvm_gmem_attach_resource(struct kvm *kvm,
+		struct inode *inode, int resource_fd)
 {
+	const struct guest_memfd_provider_operations *ops;
+	struct gmem_inode *gi = GMEM_I(inode);
 	struct file *resource_file;
-	struct inode *res_inode;
-	const struct guest_memfd_provider_operations *ops = NULL;
 	void *provider;
+	int ret;
 
 	resource_file = fget_raw(resource_fd);
 	if (!resource_file)
 		return -EBADF;
 
-	res_inode = file_inode(resource_file);
-	ops = res_inode->i_sb->s_op->gmem_provider_ops;
+	ops = resource_file->f_mapping->host->i_sb->s_op->gmem_provider_ops;
 
-	if (!ops || !ops->attach || !ops->alloc_folio) {
-		fput(resource_file);
-		return -EOPNOTSUPP;
+	ret = kvm_gmem_validate_provider(kvm, gi, ops);
+	if (ret)
+		goto out_put;
+
+	gi->device.kvm = kvm;
+	gi->device.size = i_size_read(inode);
+	/* The provider needs creation flags before attachment is published. */
+	gi->device.flags = gi->flags;
+	provider = ops->attach(resource_file, &gi->device);
+	if (IS_ERR(provider)) {
+		ret = PTR_ERR(provider);
+		goto out_put;
 	}
 
-	provider = ops->attach(resource_file);
+	gi->provider = provider;
+	gi->provider_ops = ops;
+	ret = 0;
+out_put:
 	fput(resource_file);
-	if (IS_ERR(provider))
-		return PTR_ERR(provider);
-
-	GMEM_I(inode)->provider = provider;
-	GMEM_I(inode)->provider_ops = ops;
-
-	return 0;
+	return ret;
 }
 
 static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags, int resource_fd)
@@ -1072,7 +1150,7 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags, int resour
 		goto err_inode;
 
 	if (flags & GUEST_MEMFD_FLAG_USE_RESOURCE) {
-		err = kvm_gmem_attach_resource(inode, resource_fd);
+		err = kvm_gmem_attach_resource(kvm, inode, resource_fd);
 		if (err)
 			goto err_inode;
 	}
@@ -1125,6 +1203,7 @@ int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args)
 int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot,
 		  unsigned int fd, uoff_t offset)
 {
+	struct guest_memfd_device_binding binding = {};
 	uoff_t size = slot->npages << PAGE_SHIFT;
 	unsigned long start, end;
 	struct gmem_file *f;
@@ -1140,16 +1219,16 @@ int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot,
 		return -EBADF;
 
 	if (file->f_op != &kvm_gmem_fops)
-		goto err;
+		goto out_put;
 
 	f = file->private_data;
 	if (f->kvm != kvm)
-		goto err;
+		goto out_put;
 
 	inode = file_inode(file);
 
 	if (!PAGE_ALIGNED(offset) || offset + size > i_size_read(inode))
-		goto err;
+		goto out_put;
 
 	filemap_invalidate_lock(inode->i_mapping);
 
@@ -1159,9 +1238,23 @@ int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot,
 	if (!xa_empty(&f->bindings) &&
 	    xa_find(&f->bindings, &start, end - 1, XA_PRESENT)) {
 		r = -EEXIST;
-		filemap_invalidate_unlock(inode->i_mapping);
-		goto err;
+		goto out_unlock;
+	}
+
+	if (gmem_provider_has_pfns(GMEM_I(inode))) {
+		if (slot->flags & (KVM_MEM_LOG_DIRTY_PAGES | KVM_MEM_READONLY)) {
+			r = -EINVAL;
+			goto out_unlock;
+		}
+		r = GMEM_I(inode)->provider_ops->bind(GMEM_I(inode)->provider, offset,
+						      size, &binding);
+		if (r)
+			goto out_unlock;
+		slot->flags |= KVM_MEMSLOT_GMEM_ONLY | KVM_MEMSLOT_GMEM_DEVICE;
 	}
+	r = xa_err(xa_store_range(&f->bindings, start, end - 1, slot, GFP_KERNEL));
+	if (r)
+		goto out_unlock;
 
 	/*
 	 * memslots of flag KVM_MEM_GUEST_MEMFD are immutable to change, so
@@ -1170,19 +1263,20 @@ int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot,
 	 */
 	WRITE_ONCE(slot->gmem.file, file);
 	slot->gmem.pgoff = start;
+	if (slot->flags & KVM_MEMSLOT_GMEM_DEVICE)
+		kvm_arch_gmem_device_bind(slot, &binding);
 	if (gmem_in_place_conversion || kvm_gmem_supports_mmap(inode))
 		slot->flags |= KVM_MEMSLOT_GMEM_ONLY;
 
-	xa_store_range(&f->bindings, start, end - 1, slot, GFP_KERNEL);
+	r = 0;
+out_unlock:
 	filemap_invalidate_unlock(inode->i_mapping);
-
+out_put:
 	/*
 	 * Drop the reference to the file, even on success.  The file pins KVM,
 	 * not the other way 'round.  Active bindings are invalidated if the
 	 * file is closed before memslots are destroyed.
 	 */
-	r = 0;
-err:
 	fput(file);
 	return r;
 }
@@ -1289,6 +1383,21 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 
 	filemap_invalidate_lock_shared(file_inode(file)->i_mapping);
 
+	if (gmem_provider_has_pfns(GMEM_I(file_inode(file)))) {
+		struct gmem_inode *gi = GMEM_I(file_inode(file));
+		unsigned long device_pfn;
+
+		*max_order = 0;
+		if (kvm_gmem_is_private_mem(file_inode(file), index))
+			r = -EAGAIN;
+		else
+			r = gi->provider_ops->get_pfn(gi->provider,
+						(u64)index << PAGE_SHIFT, &device_pfn);
+		if (!r)
+			*pfn = device_pfn;
+		goto out;
+	}
+
 	folio = __kvm_gmem_get_pfn(file, slot, index, pfn, max_order);
 	if (IS_ERR(folio)) {
 		r = PTR_ERR(folio);
-- 
2.43.0

[RFC PATCH v2 02/13] KVM: guest_memfd: Support private/shared conversion of device memory

[Aneesh Kumar K.V (Arm)]

Allow architectures to convert device-backed guest_memfd ranges to
private memory. Coordinate conversion with the device provider and
update guest_memfd attributes only after the architecture operation
succeeds.

Hold the inode invalidate lock across conversion and the attribute
update. Provider callbacks allow a device conversion lock to cover the
same operation, with the inode lock acquired first. This prevents faults
and competing device operations from observing an incomplete conversion.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 include/linux/guest_memfd.h |  12 +++
 include/linux/kvm_host.h    |   5 +
 virt/kvm/guest_memfd.c      | 201 +++++++++++++++++++++++++++++++++++-
 3 files changed, 213 insertions(+), 5 deletions(-)

diff --git a/include/linux/guest_memfd.h b/include/linux/guest_memfd.h
index 1bc9a23afa4e..7ae8df07a26e 100644
--- a/include/linux/guest_memfd.h
+++ b/include/linux/guest_memfd.h
@@ -9,6 +9,12 @@ struct folio;
 struct mempolicy;
 struct kvm;
 
+struct guest_memfd_device_request {
+	u64 gpa;
+	u64 pa;
+	u64 vdev_id;
+};
+
 /* Binding data stays valid only while the provider is active. */
 struct guest_memfd_device_binding {
 	/* Architecture-specific mapping owner, e.g. an RMM vDEVICE handle. */
@@ -33,6 +39,8 @@ struct guest_memfd {
  * @invalidate_folio: Notify the folio provider before releasing a folio.
  * @bind: Validate a PFN interval and return its protected mapping owner.
  * @get_pfn: Resolve a shared resource offset without taking a folio reference.
+ * @prepare_conversion: Prepare protected PFN conversion; NULL request shares.
+ * @finish_conversion: Finish conversion after attribute publication.
  *
  * Provider operations supply exactly one of alloc_folio() and get_pfn().
  * PFN providers start shared and require the bind and conversion callbacks
@@ -47,6 +55,10 @@ struct guest_memfd_provider_operations {
 	int (*bind)(void *provider, u64 offset, u64 size,
 		struct guest_memfd_device_binding *binding);
 	int (*get_pfn)(void *provider, u64 offset, unsigned long *pfn);
+	int (*prepare_conversion)(void *provider,
+		const struct guest_memfd_device_request *req);
+	void (*finish_conversion)(void *provider,
+		const struct guest_memfd_device_request *req);
 };
 
 #endif /* _LINUX_GUEST_MEMFD_H */
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 2aa728de75e8..1602f0a711d6 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -2614,6 +2614,11 @@ bool kvm_gmem_range_has_attributes(struct kvm_memory_slot *slot, gfn_t start,
 bool kvm_arch_gmem_device_supported(struct kvm *kvm);
 void kvm_arch_gmem_device_bind(struct kvm_memory_slot *slot,
 		const struct guest_memfd_device_binding *binding);
+int kvm_arch_gmem_device_map(struct kvm *kvm,
+		const struct kvm_memory_slot *slot,
+		u64 gpa, u64 size, u64 pa);
+int kvm_arch_gmem_device_unmap(struct kvm *kvm, u64 gpa, u64 size);
+int kvm_gmem_device_map(struct kvm *kvm, u64 gpa, u64 size, u64 pa, u64 vdev_id);
 int kvm_gmem_set_attributes(struct kvm_memory_slot *slot, gfn_t start,
 			    gfn_t end, u64 attributes);
 bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm);
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 9a8b66bd623a..b2782e64e9c8 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -40,6 +40,13 @@ struct gmem_inode {
 	struct list_head gmem_file_list;
 
 	struct guest_memfd device;
+	/*
+	 * Whether the completed attribute tree contains any private device range.
+	 * Updated under the inode invalidate and provider conversion locks;
+	 * provider readers use READ_ONCE() without taking the inode lock.
+	 * This avoids reversing the inode -> vDEVICE MMIO lock order.
+	 */
+	bool device_has_private;
 	void *provider;
 	const struct guest_memfd_provider_operations *provider_ops;
 
@@ -330,6 +337,10 @@ static void kvm_gmem_invalidate_end(struct inode *inode, pgoff_t start,
 		__kvm_gmem_invalidate_end(f, start, end);
 }
 
+static int kvm_gmem_device_convert(struct inode *inode, pgoff_t start,
+		pgoff_t nr_pages, const struct guest_memfd_device_request *req,
+		const struct kvm_memory_slot *slot);
+
 static long kvm_gmem_punch_hole(struct inode *inode, loff_t offset, loff_t len)
 {
 	enum kvm_gfn_range_filter filter = kvm_gmem_get_all_gfns_filter(inode);
@@ -752,8 +763,12 @@ static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 	struct ma_state mas;
 	int r = 0;
 
-	if (gmem_provider_has_pfns(gi))
-		return -EOPNOTSUPP;
+	if (gmem_provider_has_pfns(gi)) {
+		*err_index = start;
+		if (to_private)
+			return -EPERM;
+		return kvm_gmem_device_convert(inode, start, nr_pages, NULL, NULL);
+	}
 
 	mt = &gi->attributes;
 
@@ -1039,6 +1054,179 @@ void __weak kvm_arch_gmem_device_bind(struct kvm_memory_slot *slot,
 {
 }
 
+int __weak kvm_arch_gmem_device_map(struct kvm *kvm,
+		const struct kvm_memory_slot *slot,
+		u64 gpa, u64 size, u64 pa)
+{
+	return -EOPNOTSUPP;
+}
+
+int __weak kvm_arch_gmem_device_unmap(struct kvm *kvm, u64 gpa, u64 size)
+{
+	return -EOPNOTSUPP;
+}
+
+static void kvm_gmem_device_update_private(struct gmem_inode *gi)
+{
+	MA_STATE(mas, &gi->attributes, 0, 0);
+	void *entry;
+	bool private = false;
+
+	mas_for_each(&mas, entry, ULONG_MAX) {
+		if (kvm_gmem_get_attributes(&gi->vfs_inode, entry) &
+		    KVM_MEMORY_ATTRIBUTE_PRIVATE) {
+			private = true;
+			break;
+		}
+	}
+	WRITE_ONCE(gi->device_has_private, private);
+}
+
+/* The protected request and its slot identify exactly one private range. */
+static int kvm_gmem_device_make_private(struct inode *inode,
+		const struct kvm_memory_slot *slot,
+		const struct guest_memfd_device_request *req,
+		pgoff_t nr_pages)
+{
+	return kvm_arch_gmem_device_map(GMEM_I(inode)->device.kvm, slot,
+					req->gpa, (u64)nr_pages << PAGE_SHIFT,
+					req->pa);
+}
+
+/* The inode lock protects both attributes and the offset-to-GPA bindings. */
+static int kvm_gmem_device_make_shared(struct inode *inode,
+	       pgoff_t start, pgoff_t end)
+{
+	struct gmem_inode *gi = GMEM_I(inode);
+	struct gmem_file *f;
+	void *entry;
+
+	MA_STATE(mas, &gi->attributes, start, start);
+
+	mas_for_each(&mas, entry, end - 1) {
+		pgoff_t first = max(start, mas.index);
+		pgoff_t last = min(end, mas.last + 1);
+
+		if (!(kvm_gmem_get_attributes(inode, entry) &
+		      KVM_MEMORY_ATTRIBUTE_PRIVATE))
+			continue;
+
+		while (first < last) {
+			struct kvm_memory_slot *slot = NULL;
+			pgoff_t high;
+			u64 gpa;
+			int ret;
+
+			kvm_gmem_for_each_file(f, inode) {
+				slot = xa_load(&f->bindings, first);
+				if (slot)
+					break;
+			}
+			if (!slot)
+				return -EINVAL;
+
+			high = min(last, slot->gmem.pgoff + slot->npages);
+			gpa = (slot->base_gfn + first - slot->gmem.pgoff) << PAGE_SHIFT;
+			ret = kvm_arch_gmem_device_unmap(gi->device.kvm, gpa,
+						      (high - first) << PAGE_SHIFT);
+			if (ret)
+				return ret;
+
+			first = high;
+		}
+	}
+	return 0;
+}
+
+static int kvm_gmem_device_convert(struct inode *inode, pgoff_t start,
+		pgoff_t nr_pages, const struct guest_memfd_device_request *req,
+		const struct kvm_memory_slot *slot)
+{
+	struct gmem_inode *gi = GMEM_I(inode);
+	u64 attrs = req ? KVM_MEMORY_ATTRIBUTE_PRIVATE : 0;
+	bool prepared = false;
+	int ret;
+
+	MA_STATE(mas, &gi->attributes, start, start + nr_pages - 1);
+
+	filemap_invalidate_lock(inode->i_mapping);
+	if (__kvm_gmem_range_has_attributes(inode, start, nr_pages, attrs)) {
+		ret = 0;
+		goto out;
+	}
+
+	if (req && !__kvm_gmem_range_has_attributes(inode, start, nr_pages, 0)) {
+		ret = -EEXIST;
+		goto out;
+	}
+
+	ret = kvm_gmem_mas_preallocate(&mas, attrs, start, nr_pages);
+	if (ret)
+		goto out;
+
+	kvm_gmem_invalidate_start(inode, start, start + nr_pages, KVM_FILTER_SHARED);
+	if (req && READ_ONCE(gi->device.kvm->vm_dead))
+		ret = -EIO;
+	else
+		ret = gi->provider_ops->prepare_conversion(gi->provider, req);
+	if (!ret) {
+		prepared = true;
+		if (req)
+			ret = kvm_gmem_device_make_private(inode, slot, req, nr_pages);
+		else
+			ret = kvm_gmem_device_make_shared(inode, start, start + nr_pages);
+	}
+	if (!ret) {
+		mas_store_prealloc(&mas, xa_mk_value(attrs));
+		kvm_gmem_device_update_private(gi);
+	} else {
+		mas_destroy(&mas);
+	}
+
+	kvm_gmem_invalidate_end(inode, start, start + nr_pages);
+	if (prepared)
+		gi->provider_ops->finish_conversion(gi->provider, req);
+out:
+	filemap_invalidate_unlock(inode->i_mapping);
+	return ret;
+}
+
+int kvm_gmem_device_map(struct kvm *kvm, u64 gpa, u64 size, u64 pa, u64 vdev_id)
+{
+	struct guest_memfd_device_request req = { gpa, pa, vdev_id };
+	struct kvm_memory_slot *slot;
+	gfn_t gfn = gpa >> PAGE_SHIFT;
+	struct file *file;
+	int ret;
+
+	if (!size)
+		return -EINVAL;
+
+	if (!PAGE_ALIGNED(gpa | size | pa))
+		return -EINVAL;
+
+	/* Caller holds SRCU across the pending architecture request. */
+	slot = gfn_to_memslot(kvm, gfn);
+	if (!slot || (slot->flags & KVM_MEMSLOT_INVALID))
+		return -EINVAL;
+
+	if (!(slot->flags & KVM_MEMSLOT_GMEM_DEVICE))
+		return -EINVAL;
+
+	if ((size >> PAGE_SHIFT) > slot->npages - (gfn - slot->base_gfn))
+		return -EINVAL;
+
+	file = kvm_gmem_get_file(slot);
+	if (!file)
+		return -ENOENT;
+	ret = kvm_gmem_device_convert(file_inode(file),
+				      kvm_gmem_get_index(slot, gfn),
+				      size >> PAGE_SHIFT, &req, slot);
+	fput(file);
+	return ret;
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_gmem_device_map);
+
 static int kvm_gmem_validate_provider(struct kvm *kvm,
 		struct gmem_inode *gi,
 		const struct guest_memfd_provider_operations *ops)
@@ -1054,13 +1242,15 @@ static int kvm_gmem_validate_provider(struct kvm *kvm,
 	/* Folio providers use the core's binding and conversion paths. */
 	if (!ops->get_pfn) {
 		/* PFN-specific callbacks are invalid for folio providers. */
-		if (ops->bind)
+		if (ops->bind || ops->prepare_conversion ||
+		    ops->finish_conversion)
 			return -EOPNOTSUPP;
 		return 0;
 	}
 
-	/* PFN providers must support binding. */
-	if (!ops->bind)
+	/* PFN providers must support binding and conversion. */
+	if (!ops->bind || !ops->prepare_conversion ||
+	    !ops->finish_conversion)
 		return -EOPNOTSUPP;
 
 	/* PFN providers do not manage folios. */
@@ -1552,6 +1742,7 @@ static struct inode *kvm_gmem_alloc_inode(struct super_block *sb)
 	gi->flags = 0;
 	gi->provider = NULL;
 	gi->provider_ops = NULL;
+	gi->device_has_private = false;
 	INIT_LIST_HEAD(&gi->gmem_file_list);
 	return &gi->vfs_inode;
 }
-- 
2.43.0

[RFC PATCH v2 03/13] iommufd: Support MMIO provider attachment to vDEVICEs

[Aneesh Kumar K.V (Arm)]

Allow a device MMIO provider to attach to the existing vDEVICE after the
backend validates its association with the VM. Keep the vDEVICE and
IOMMUFD context alive until the provider detaches.

Add a per-vDEVICE MMIO mutex to serialize provider attachment and
detachment, and allow providers and backends to use the same lock for
memory conversion and device state changes. Provide a callback to query
whether private MMIO ranges remain.

Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Kevin Tian <kevin.tian@intel.com>
Cc: "Joerg Roedel (AMD)" <joro@8bytes.org>
Cc: Will Deacon <will@kernel.org>
Cc: Robin Murphy <robin.murphy@arm.com>
Cc: iommu@lists.linux.dev
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 drivers/iommu/iommufd/viommu.c | 86 ++++++++++++++++++++++++++++++++++
 include/linux/iommufd.h        | 26 ++++++++++
 2 files changed, 112 insertions(+)

diff --git a/drivers/iommu/iommufd/viommu.c b/drivers/iommu/iommufd/viommu.c
index 9155d0dcb4b4..5561d735616c 100644
--- a/drivers/iommu/iommufd/viommu.c
+++ b/drivers/iommu/iommufd/viommu.c
@@ -250,6 +250,7 @@ int iommufd_vdevice_alloc_ioctl(struct iommufd_ucmd *ucmd)
 		goto out_unlock_igroup;
 	}
 
+	mutex_init(&vdev->mmio_lock);
 	vdev->virt_id = virt_id;
 	vdev->viommu = viommu;
 	refcount_inc(&viommu->obj.users);
@@ -487,3 +488,88 @@ int iommufd_hw_queue_alloc_ioctl(struct iommufd_ucmd *ucmd)
 	iommufd_put_object(ucmd->ictx, &viommu->obj);
 	return rc;
 }
+
+struct iommufd_vdevice *
+iommufd_device_attach_mmio_provider(struct iommufd_device *idev,
+		struct kvm *kvm, void *data,
+		bool (*has_private_mmio)(void *),
+		unsigned long *owner)
+{
+	struct iommufd_vdevice *vdev;
+	int ret;
+
+	if (!idev)
+		return ERR_PTR(-ENODEV);
+
+	guard(mutex)(&idev->igroup->lock);
+	if (idev->destroying)
+		return ERR_PTR(-EOPNOTSUPP);
+
+	vdev = idev->vdev;
+	if (!vdev)
+		return ERR_PTR(-EOPNOTSUPP);
+
+	if (!vdev->viommu->ops ||
+	    !vdev->viommu->ops->vdevice_get_mmio_owner)
+		return ERR_PTR(-EOPNOTSUPP);
+
+	guard(mutex)(&vdev->mmio_lock);
+	if (vdev->mmio_provider_data)
+		return ERR_PTR(-EBUSY);
+
+	ret = vdev->viommu->ops->vdevice_get_mmio_owner(vdev, kvm, owner);
+	if (ret)
+		return ERR_PTR(ret);
+
+	/* IOMMU_DESTROY must wait for the attached gmem provider to detach. */
+	ret = iommufd_try_inc_users(idev->ictx, &vdev->obj);
+	if (ret)
+		return ERR_PTR(ret);
+
+	iommufd_ctx_get(idev->ictx);
+	vdev->mmio_provider_data = data;
+	vdev->has_private_mmio = has_private_mmio;
+	return vdev;
+}
+EXPORT_SYMBOL_NS_GPL(iommufd_device_attach_mmio_provider, "IOMMUFD");
+
+void iommufd_vdevice_detach_mmio_provider(struct iommufd_vdevice *vdev)
+{
+	struct iommufd_ctx *ictx = vdev->viommu->ictx;
+
+	scoped_guard(mutex, &vdev->mmio_lock) {
+		WARN_ON(iommufd_vdevice_has_private_mmio(vdev));
+		vdev->mmio_provider_data = NULL;
+		vdev->has_private_mmio = NULL;
+	}
+	refcount_dec(&vdev->obj.users);
+	iommufd_ctx_put(ictx);
+}
+EXPORT_SYMBOL_NS_GPL(iommufd_vdevice_detach_mmio_provider, "IOMMUFD");
+
+void iommufd_vdevice_mmio_lock(struct iommufd_vdevice *vdev)
+{
+	mutex_lock(&vdev->mmio_lock);
+}
+EXPORT_SYMBOL_NS_GPL(iommufd_vdevice_mmio_lock, "IOMMUFD");
+
+void iommufd_vdevice_mmio_unlock(struct iommufd_vdevice *vdev)
+{
+	mutex_unlock(&vdev->mmio_lock);
+}
+EXPORT_SYMBOL_NS_GPL(iommufd_vdevice_mmio_unlock, "IOMMUFD");
+
+/**
+ * iommufd_vdevice_has_private_mmio() - check for private MMIO ranges
+ * @vdev: vDEVICE whose private MMIO state is being checked
+ *
+ * Return: true if the attached provider has committed private ranges.
+ */
+bool iommufd_vdevice_has_private_mmio(struct iommufd_vdevice *vdev)
+{
+	lockdep_assert_held(&vdev->mmio_lock);
+	if (!vdev->has_private_mmio)
+		return false;
+	return vdev->has_private_mmio(vdev->mmio_provider_data);
+}
+EXPORT_SYMBOL_NS_GPL(iommufd_vdevice_has_private_mmio, "IOMMUFD");
diff --git a/include/linux/iommufd.h b/include/linux/iommufd.h
index 2e67846d6f35..5fbafbdb4d88 100644
--- a/include/linux/iommufd.h
+++ b/include/linux/iommufd.h
@@ -7,10 +7,12 @@
 #define __LINUX_IOMMUFD_H
 
 #include <linux/bits.h>
+#include <linux/cleanup.h>
 #include <linux/err.h>
 #include <linux/errno.h>
 #include <linux/iommu.h>
 #include <linux/refcount.h>
+#include <linux/mutex.h>
 #include <linux/types.h>
 #include <linux/xarray.h>
 #include <uapi/linux/iommufd.h>
@@ -26,6 +28,7 @@ struct iommufd_device;
 struct iommufd_viommu_ops;
 struct tsm_dev;
 struct page;
+struct kvm;
 struct tsm_guest_req_info;
 
 enum iommufd_object_type {
@@ -73,6 +76,19 @@ int iommufd_device_replace(struct iommufd_device *idev, ioasid_t pasid,
 			   u32 *pt_id);
 void iommufd_device_detach(struct iommufd_device *idev, ioasid_t pasid);
 
+/* The returned opaque vDEVICE pins its IOMMUFD context and physical owner. */
+struct iommufd_vdevice *
+iommufd_device_attach_mmio_provider(struct iommufd_device *idev,
+		struct kvm *kvm, void *data,
+		bool (*has_private_mmio)(void *),
+		unsigned long *owner);
+void iommufd_vdevice_detach_mmio_provider(struct iommufd_vdevice *vdev);
+void iommufd_vdevice_mmio_lock(struct iommufd_vdevice *vdev);
+void iommufd_vdevice_mmio_unlock(struct iommufd_vdevice *vdev);
+DEFINE_GUARD(iommufd_vdevice_mmio, struct iommufd_vdevice *,
+	     iommufd_vdevice_mmio_lock(_T), iommufd_vdevice_mmio_unlock(_T))
+bool iommufd_vdevice_has_private_mmio(struct iommufd_vdevice *vdev);
+
 struct iommufd_ctx *iommufd_device_to_ictx(struct iommufd_device *idev);
 u32 iommufd_device_to_id(struct iommufd_device *idev);
 
@@ -130,6 +146,12 @@ struct iommufd_vdevice {
 	 */
 	u64 virt_id;
 
+	/* Serializes provider attachment and conversion. */
+	struct mutex mmio_lock;
+	/* Attached provider and callbacks, protected by mmio_lock until detach. */
+	void *mmio_provider_data;
+	bool (*has_private_mmio)(void *data);
+
 	/* Guest TSM requests accepted by this vdevice; set by vdevice_init(). */
 	u64 tsm_req_op_mask;
 	u32 tsm_tvm_arch;
@@ -187,6 +209,7 @@ struct iommufd_hw_queue {
  *                include/uapi/linux/iommufd.h)
  *                If driver has a deinit function to revert what vdevice_init op
  *                does, it should set it to the @vdev->destroy function pointer
+ * @vdevice_get_mmio_owner: Validate the VM and return its MMIO owner.
  * @vdevice_tsm_req: Forward a guest TSM request to a driver-owned vDEVICE
  * @get_hw_queue_size: Get the size of a driver-defined HW queue structure for a
  *                     given @viommu corresponding to @queue_type. Driver should
@@ -218,6 +241,9 @@ struct iommufd_viommu_ops {
 		const struct iommu_user_data *user_data);
 	int (*cache_invalidate)(struct iommufd_viommu *viommu,
 				struct iommu_user_data_array *array);
+	/* Owner lookup runs with vdevice->mmio_lock held. */
+	int (*vdevice_get_mmio_owner)(struct iommufd_vdevice *vdev,
+		struct kvm *kvm, unsigned long *owner);
 	const size_t vdevice_size;
 	int (*vdevice_init)(struct iommufd_vdevice *vdev);
 	ssize_t (*vdevice_tsm_req)(struct iommufd_vdevice *vdev,
-- 
2.43.0

[RFC PATCH v2 04/13] KVM: guest_memfd: Add device provider access callbacks

[Aneesh Kumar K.V (Arm)]

Expose callbacks for device providers to check whether a range is shared,
query whether private ranges remain, and invalidate shared guest mappings.

Use a nonblocking acquisition of the inode invalidate lock for host
access checks, avoiding deadlock with conversion while a provider holds
its memory lock. Invalidate shared mappings under SRCU and the KVM MMU
lock, advancing the MMU invalidation sequence so concurrent faults retry.

Initialize callback state before attaching the provider and publish the
guest_memfd file after creation completes.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 include/linux/guest_memfd.h |   6 ++
 virt/kvm/guest_memfd.c      | 117 ++++++++++++++++++++++++++++++++++++
 2 files changed, 123 insertions(+)

diff --git a/include/linux/guest_memfd.h b/include/linux/guest_memfd.h
index 7ae8df07a26e..f0c62889f823 100644
--- a/include/linux/guest_memfd.h
+++ b/include/linux/guest_memfd.h
@@ -26,6 +26,12 @@ struct guest_memfd {
 	struct kvm *kvm;
 	u64 size;
 	u64 flags;
+	struct file *gmem_file;
+
+	/* Revoke shared guest translations synchronously. */
+	void (*invalidate)(struct guest_memfd *gmem);
+	bool (*has_private)(struct guest_memfd *gmem);
+	bool (*host_accessible)(struct guest_memfd *gmem, u64 offset, u64 size);
 };
 
 /**
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index b2782e64e9c8..d2b1e2e9dfee 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -1066,6 +1066,118 @@ int __weak kvm_arch_gmem_device_unmap(struct kvm *kvm, u64 gpa, u64 size)
 	return -EOPNOTSUPP;
 }
 
+static bool gmem_device_host_accessible(struct guest_memfd *gdev,
+		u64 offset, u64 size)
+{
+	struct file *file = READ_ONCE(gdev->gmem_file);
+	struct inode *inode;
+	pgoff_t first, last;
+	bool shared;
+
+	if (!file)
+		return false;
+
+	/* Unwrapped BAR offsets have no private guest_memfd mappings. */
+	if (offset >= gdev->size)
+		return true;
+
+	size = min(size, gdev->size - offset);
+	if (!size)
+		return false;
+
+	first = offset >> PAGE_SHIFT;
+	last = ((offset + size - 1) >> PAGE_SHIFT) + 1;
+	inode = file_inode(file);
+	/*
+	 * Keep guest_memfd attributes stable against conversion while checking
+	 * host access. This check only reads attributes, so a shared lock lets
+	 * other access checks run concurrently.
+	 *
+	 * VFIO callers hold memory_lock, but conversion takes the invalidate
+	 * lock exclusively before taking memory_lock for writing.
+	 * Waiting here could deadlock with conversion. Try the shared lock and
+	 * deny access if it cannot be acquired.
+	 */
+	if (!filemap_invalidate_trylock_shared(inode->i_mapping))
+		return false;
+
+	shared = __kvm_gmem_range_has_attributes(inode, first, last - first, 0);
+	filemap_invalidate_unlock_shared(inode->i_mapping);
+	return shared;
+}
+
+/**
+ * gmem_device_invalidate() - invalidate shared guest mappings of device memory
+ * @gdev: device guest_memfd whose provider is revoking access
+ *
+ * VFIO invokes this callback when zapping BAR mappings. Removing host BAR
+ * mappings alone leaves guest stage-2 mappings intact, so unmap the shared
+ * ranges in memslots backed by this guest_memfd and flush guest TLBs as needed.
+ * Advance the MMU invalidation sequence so faults that obtained a provider PFN
+ * before revocation retry instead of installing a stale mapping.
+ *
+ * SRCU protects memslot traversal, and the KVM MMU lock serializes unmapping
+ * with guest faults. Do not acquire the inode invalidate lock: conversion may
+ * already hold it, and VFIO callers hold memory_lock in the reverse order.
+ *
+ * Caller holds the provider memory lock to exclude new provider PFN
+ * lookups during invalidation.
+ */
+static void gmem_device_invalidate(struct guest_memfd *gdev)
+{
+	struct file *file = gdev->gmem_file;
+	struct kvm *kvm = gdev->kvm;
+	struct kvm_memory_slot *slot;
+	struct kvm_memslots *slots;
+	int srcu_idx, bkt, as;
+	bool flush = false, found_memslot = false;
+
+	if (!file)
+		return;
+
+	srcu_idx = srcu_read_lock(&kvm->srcu);
+	KVM_MMU_LOCK(kvm);
+
+	for (as = 0; as < kvm_arch_nr_memslot_as_ids(kvm); as++) {
+		slots = __kvm_memslots(kvm, as);
+		kvm_for_each_memslot(slot, bkt, slots) {
+			struct kvm_gfn_range range = {
+				.slot = slot,
+				.start = slot->base_gfn,
+				.end = slot->base_gfn + slot->npages,
+				.attr_filter = KVM_FILTER_SHARED,
+				.may_block = false,
+			};
+
+			if (READ_ONCE(slot->gmem.file) != file)
+				continue;
+
+			if (!found_memslot) {
+				found_memslot = true;
+				kvm_mmu_invalidate_start(kvm);
+			}
+
+			flush |= kvm_mmu_unmap_gfn_range(kvm, &range);
+		}
+	}
+	if (flush)
+		kvm_flush_remote_tlbs(kvm);
+
+	if (found_memslot)
+		kvm_mmu_invalidate_end(kvm);
+
+	KVM_MMU_UNLOCK(kvm);
+	srcu_read_unlock(&kvm->srcu, srcu_idx);
+}
+
+/* The attribute tree owns mapping state; this summary is read under provider locks. */
+static bool gmem_device_has_private(struct guest_memfd *gdev)
+{
+	struct gmem_inode *gi = container_of(gdev, struct gmem_inode, device);
+
+	return READ_ONCE(gi->device_has_private);
+}
+
 static void kvm_gmem_device_update_private(struct gmem_inode *gi)
 {
 	MA_STATE(mas, &gi->attributes, 0, 0);
@@ -1291,6 +1403,9 @@ static int kvm_gmem_attach_resource(struct kvm *kvm,
 	gi->device.size = i_size_read(inode);
 	/* The provider needs creation flags before attachment is published. */
 	gi->device.flags = gi->flags;
+	gi->device.invalidate = gmem_device_invalidate;
+	gi->device.host_accessible = gmem_device_host_accessible;
+	gi->device.has_private = gmem_device_has_private;
 	provider = ops->attach(resource_file, &gi->device);
 	if (IS_ERR(provider)) {
 		ret = PTR_ERR(provider);
@@ -1359,6 +1474,7 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags, int resour
 	xa_init(&f->bindings);
 	list_add(&f->entry, &GMEM_I(inode)->gmem_file_list);
 
+	WRITE_ONCE(GMEM_I(inode)->device.gmem_file, file);
 	fd_install(fd, file);
 	return fd;
 
@@ -1743,6 +1859,7 @@ static struct inode *kvm_gmem_alloc_inode(struct super_block *sb)
 	gi->provider = NULL;
 	gi->provider_ops = NULL;
 	gi->device_has_private = false;
+	memset(&gi->device, 0, sizeof(gi->device));
 	INIT_LIST_HEAD(&gi->gmem_file_list);
 	return &gi->vfs_inode;
 }
-- 
2.43.0

[RFC PATCH v2 05/13] vfio/pci: Track userspace BAR mappings

[Aneesh Kumar K.V (Arm)]

Track userspace mappings of PCI BARs so a later patch can reject
guest_memfd attachment while such mappings exist.

Count the initial mmap explicitly because vm_ops->open() is not called
for it. Use the VMA open and close callbacks to track duplication and
unmapping, keeping the count nonzero until the last mapping is unmapped.

Cc: Alex Williamson <alex@shazbot.org>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 drivers/vfio/pci/vfio_pci_core.c | 19 +++++++++++++++++++
 include/linux/vfio_pci_core.h    |  2 ++
 2 files changed, 21 insertions(+)

diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index 6757054e9d87..795b4ab1174e 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -1806,7 +1806,23 @@ static vm_fault_t vfio_pci_mmap_page_fault(struct vm_fault *vmf)
 	return vfio_pci_mmap_huge_fault(vmf, 0);
 }
 
+static void vfio_pci_mmap_open(struct vm_area_struct *vma)
+{
+	struct vfio_pci_core_device *vdev = vma->vm_private_data;
+
+	atomic_inc(&vdev->mmap_count);
+}
+
+static void vfio_pci_mmap_close(struct vm_area_struct *vma)
+{
+	struct vfio_pci_core_device *vdev = vma->vm_private_data;
+
+	atomic_dec(&vdev->mmap_count);
+}
+
 static const struct vm_operations_struct vfio_pci_mmap_ops = {
+	.open = vfio_pci_mmap_open,
+	.close = vfio_pci_mmap_close,
 	.fault = vfio_pci_mmap_page_fault,
 #ifdef CONFIG_ARCH_SUPPORTS_HUGE_PFNMAP
 	.huge_fault = vfio_pci_mmap_huge_fault,
@@ -1888,6 +1904,8 @@ int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct vm_area_struct *vma
 	vm_flags_set(vma, VM_ALLOW_ANY_UNCACHED | VM_IO | VM_PFNMAP |
 			VM_DONTEXPAND | VM_DONTDUMP);
 	vma->vm_ops = &vfio_pci_mmap_ops;
+	/* Initial mmap does not invoke vm_ops->open(). */
+	vfio_pci_mmap_open(vma);
 
 	return 0;
 }
@@ -2197,6 +2215,7 @@ int vfio_pci_core_init_dev(struct vfio_device *core_vdev)
 		return ret;
 	INIT_LIST_HEAD(&vdev->dmabufs);
 	init_rwsem(&vdev->memory_lock);
+	atomic_set(&vdev->mmap_count, 0);
 	xa_init(&vdev->ctx);
 
 	return 0;
diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index 9a1674c152aa..dc15b462126e 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -8,6 +8,7 @@
  * Author: Tom Lyon, pugs@cisco.com
  */
 
+#include <linux/atomic.h>
 #include <linux/mutex.h>
 #include <linux/pci.h>
 #include <linux/vfio.h>
@@ -148,6 +149,7 @@ struct vfio_pci_core_device {
 	struct vfio_pci_core_device	*sriov_pf_core_dev;
 	struct notifier_block	nb;
 	struct rw_semaphore	memory_lock;
+	atomic_t		mmap_count;
 	struct list_head	dmabufs;
 };
 
-- 
2.43.0

[RFC PATCH v2 06/13] vfio: Dispatch guest_memfd provider operations through the filesystem

[Aneesh Kumar K.V (Arm)]

Expose guest_memfd provider operations through the VFIO mapping
superblock and forward them to the device driver. Validate the driver
callbacks before attachment and keep the source file pinned through
provider release.

Cc: Alex Williamson <alex@shazbot.org>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 drivers/vfio/vfio_main.c | 115 ++++++++++++++++++++++++++++++++++++++-
 include/linux/vfio.h     |   4 ++
 2 files changed, 118 insertions(+), 1 deletion(-)

diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
index ed96acfa8635..2051e14ac616 100644
--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -14,8 +14,10 @@
 #include <linux/compat.h>
 #include <linux/device.h>
 #include <linux/fs.h>
+#include <linux/guest_memfd.h>
 #include <linux/idr.h>
 #include <linux/iommu.h>
+#include <linux/kvm.h>
 #if IS_ENABLED(CONFIG_KVM)
 #include <linux/kvm_host.h>
 #endif
@@ -248,9 +250,120 @@ struct vfio_device *_vfio_alloc_device(size_t size, struct device *dev,
 }
 EXPORT_SYMBOL_GPL(_vfio_alloc_device);
 
+struct vfio_gmem_provider {
+	const struct guest_memfd_provider_operations *ops;
+	void *data;
+	struct file *file;
+};
+
+static void *vfio_gmem_provider_attach(struct file *file,
+		struct guest_memfd *gmem)
+{
+	const struct guest_memfd_provider_operations *ops;
+	struct vfio_gmem_provider *provider;
+	struct vfio_device_file *df;
+	void *data;
+
+	/* Reject other file types before interpreting their private_data. */
+	if (file->f_op != &vfio_device_fops)
+		return ERR_PTR(-EOPNOTSUPP);
+
+	df = file->private_data;
+	ops = df->device->ops->gmem_provider_ops;
+	/* Every provider must support attachment and release. */
+	if (!ops || !ops->attach || !ops->release)
+		return ERR_PTR(-EOPNOTSUPP);
+
+	/* VFIO requires PFN lookup, binding and conversion. */
+	if (!ops->get_pfn || !ops->bind || !ops->prepare_conversion ||
+	    !ops->finish_conversion)
+		return ERR_PTR(-EOPNOTSUPP);
+
+	/* VFIO provides PFNs, not folios. */
+	if (ops->alloc_folio || ops->invalidate_folio)
+		return ERR_PTR(-EOPNOTSUPP);
+
+	provider = kzalloc_obj(*provider);
+	if (!provider)
+		return ERR_PTR(-ENOMEM);
+
+	data = ops->attach(file, gmem);
+	if (IS_ERR(data)) {
+		kfree(provider);
+		return data;
+	}
+
+	provider->ops = ops;
+	provider->data = data;
+	/* Pin the driver operations through release(), including its return. */
+	provider->file = get_file(file);
+	return provider;
+}
+
+static void vfio_gmem_provider_release(void *data)
+{
+	struct vfio_gmem_provider *provider = data;
+
+	provider->ops->release(provider->data);
+	fput(provider->file);
+	kfree(provider);
+}
+
+static int vfio_gmem_provider_bind(void *data, u64 offset, u64 size,
+				   struct guest_memfd_device_binding *binding)
+{
+	struct vfio_gmem_provider *provider = data;
+
+	return provider->ops->bind(provider->data, offset, size, binding);
+}
+
+static int vfio_gmem_provider_get_pfn(void *data, u64 offset, unsigned long *pfn)
+{
+	struct vfio_gmem_provider *provider = data;
+
+	return provider->ops->get_pfn(provider->data, offset, pfn);
+}
+
+static int vfio_gmem_provider_prepare_conversion(
+		void *context, const struct guest_memfd_device_request *req)
+{
+	struct vfio_gmem_provider *provider = context;
+
+	return provider->ops->prepare_conversion(provider->data, req);
+}
+
+static void vfio_gmem_provider_finish_conversion(
+		void *context, const struct guest_memfd_device_request *req)
+{
+	struct vfio_gmem_provider *provider = context;
+
+	provider->ops->finish_conversion(provider->data, req);
+}
+
+
+
+static const struct guest_memfd_provider_operations vfio_gmem_provider_ops = {
+	.attach = vfio_gmem_provider_attach,
+	.release = vfio_gmem_provider_release,
+	.bind = vfio_gmem_provider_bind,
+	.get_pfn = vfio_gmem_provider_get_pfn,
+	.prepare_conversion = vfio_gmem_provider_prepare_conversion,
+	.finish_conversion = vfio_gmem_provider_finish_conversion,
+};
+
+static const struct super_operations vfio_fs_super_ops = {
+	.statfs = simple_statfs,
+	.gmem_provider_ops = &vfio_gmem_provider_ops,
+};
+
 static int vfio_fs_init_fs_context(struct fs_context *fc)
 {
-	return init_pseudo(fc, VFIO_MAGIC) ? 0 : -ENOMEM;
+	struct pseudo_fs_context *ctx = init_pseudo(fc, VFIO_MAGIC);
+
+	if (!ctx)
+		return -ENOMEM;
+	ctx->ops = &vfio_fs_super_ops;
+	return 0;
 }
 
 static struct file_system_type vfio_fs_type = {
diff --git a/include/linux/vfio.h b/include/linux/vfio.h
index 0cc91c6f96d2..f33573f99048 100644
--- a/include/linux/vfio.h
+++ b/include/linux/vfio.h
@@ -81,6 +81,8 @@ struct vfio_device {
 	struct dentry *debug_root;
 };
 
+struct guest_memfd_provider_operations;
+
 /**
  * struct vfio_device_ops - VFIO bus driver device callbacks
  *
@@ -111,9 +113,11 @@ struct vfio_device {
  *         if none was provided.
  * @dma_unmap: Called when userspace unmaps IOVA from the container
  *             this device is attached to.
+ * @gmem_provider_ops: Optional driver guest_memfd resource operations
  * @device_feature: Optional, fill in the VFIO_DEVICE_FEATURE ioctl
  */
 struct vfio_device_ops {
+	const struct guest_memfd_provider_operations *gmem_provider_ops;
 	char	*name;
 	int	(*init)(struct vfio_device *vdev);
 	void	(*release)(struct vfio_device *vdev);
-- 
2.43.0

[RFC PATCH v2 07/13] vfio/pci: Restrict host BAR access for guest_memfd providers

[Aneesh Kumar K.V (Arm)]

Add the access and invalidation helpers needed by a PCI guest_memfd
provider. Check shared-range attributes for BAR read/write and ioeventfd
access, and revoke shared guest mappings when VFIO changes device state.

Reject new VFIO mmap and dma-buf exports while a provider is attached.

Cc: Alex Williamson <alex@shazbot.org>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 drivers/vfio/pci/Kconfig                |  11 +++
 drivers/vfio/pci/Makefile               |   1 +
 drivers/vfio/pci/vfio_pci_core.c        |   9 ++-
 drivers/vfio/pci/vfio_pci_dmabuf.c      |  10 +++
 drivers/vfio/pci/vfio_pci_gmem.h        |  15 ++++
 drivers/vfio/pci/vfio_pci_gmem_access.c | 101 ++++++++++++++++++++++++
 drivers/vfio/pci/vfio_pci_rdwr.c        |  13 ++-
 include/linux/vfio_pci_core.h           |  22 ++++++
 8 files changed, 178 insertions(+), 4 deletions(-)
 create mode 100644 drivers/vfio/pci/vfio_pci_gmem.h
 create mode 100644 drivers/vfio/pci/vfio_pci_gmem_access.c

diff --git a/drivers/vfio/pci/Kconfig b/drivers/vfio/pci/Kconfig
index 296bf01e185e..e5dedf3e4a65 100644
--- a/drivers/vfio/pci/Kconfig
+++ b/drivers/vfio/pci/Kconfig
@@ -55,6 +55,17 @@ config VFIO_PCI_ZDEV_KVM
 
 	  To enable s390x KVM vfio-pci extensions, say Y.
 
+config VFIO_PCI_GMEM
+	bool "VFIO PCI device guest_memfd resource provider"
+	depends on VFIO_PCI && IOMMUFD && KVM
+	depends on KVM_GUEST_MEMFD
+	help
+	  Allow the VFIO PCI device fd to provide BAR PFNs to guest_memfd,
+	  including confidential-device range conversion through its
+	  existing IOMMUFD vDEVICE. This requires a capable TSM backend.
+	  Shared access is revoked before private range mapping and is
+	  restored only after private mappings and device state permit it.
+
 config VFIO_PCI_DMABUF
 	def_bool y if VFIO_PCI_CORE && PCI_P2PDMA && DMA_SHARED_BUFFER
 
diff --git a/drivers/vfio/pci/Makefile b/drivers/vfio/pci/Makefile
index 6138f1bf241d..d491b38ad9c2 100644
--- a/drivers/vfio/pci/Makefile
+++ b/drivers/vfio/pci/Makefile
@@ -3,6 +3,7 @@
 vfio-pci-core-y := vfio_pci_core.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o
 vfio-pci-core-$(CONFIG_VFIO_PCI_ZDEV_KVM) += vfio_pci_zdev.o
 vfio-pci-core-$(CONFIG_VFIO_PCI_DMABUF) += vfio_pci_dmabuf.o
+vfio-pci-core-$(CONFIG_VFIO_PCI_GMEM) += vfio_pci_gmem_access.o
 obj-$(CONFIG_VFIO_PCI_CORE) += vfio-pci-core.o
 
 vfio-pci-y := vfio_pci.o
diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index 795b4ab1174e..1258343a6832 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -1713,7 +1713,10 @@ static void vfio_pci_zap_bars(struct vfio_pci_core_device *vdev)
 	loff_t end = VFIO_PCI_INDEX_TO_OFFSET(VFIO_PCI_ROM_REGION_INDEX);
 	loff_t len = end - start;
 
-	unmap_mapping_range(core_vdev->inode->i_mapping, start, len, true);
+	/* An attached provider excludes every VFIO BAR VMA. */
+	if (!vdev->gmem)
+		unmap_mapping_range(core_vdev->inode->i_mapping, start, len, true);
+	vfio_pci_gmem_invalidate(vdev);
 }
 
 void vfio_pci_zap_and_down_write_memory_lock(struct vfio_pci_core_device *vdev)
@@ -1838,6 +1841,10 @@ int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct vm_area_struct *vma
 	u64 phys_len, req_len, pgoff, req_start;
 	void __iomem *bar_io;
 
+	guard(rwsem_read)(&vdev->memory_lock);
+	if (vdev->gmem)
+		return -EBUSY;
+
 	index = vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT);
 
 	if (index >= VFIO_PCI_NUM_REGIONS + vdev->num_regions)
diff --git a/drivers/vfio/pci/vfio_pci_dmabuf.c b/drivers/vfio/pci/vfio_pci_dmabuf.c
index c16f460c01d6..b3ae2d97629f 100644
--- a/drivers/vfio/pci/vfio_pci_dmabuf.c
+++ b/drivers/vfio/pci/vfio_pci_dmabuf.c
@@ -306,6 +306,16 @@ int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
 	/* dma_buf_put() now frees priv */
 	INIT_LIST_HEAD(&priv->dmabufs_elm);
 	down_write(&vdev->memory_lock);
+	/*
+	 * dma-buf consumers bypass guest_memfd range access checks, and their
+	 * mappings are not revoked by conversion to private memory. Exclude
+	 * exports while attached; memory_lock serializes this with attachment.
+	 */
+	if (vdev->gmem) {
+		up_write(&vdev->memory_lock);
+		dma_buf_put(priv->dmabuf);
+		return -EBUSY;
+	}
 	dma_resv_lock(priv->dmabuf->resv, NULL);
 	priv->revoked = !__vfio_pci_memory_enabled(vdev);
 	list_add_tail(&priv->dmabufs_elm, &vdev->dmabufs);
diff --git a/drivers/vfio/pci/vfio_pci_gmem.h b/drivers/vfio/pci/vfio_pci_gmem.h
new file mode 100644
index 000000000000..a8edd20357e7
--- /dev/null
+++ b/drivers/vfio/pci/vfio_pci_gmem.h
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef VFIO_PCI_GMEM_H
+#define VFIO_PCI_GMEM_H
+
+#include <linux/guest_memfd.h>
+
+struct iommufd_vdevice;
+struct vfio_pci_core_device;
+
+struct vfio_pci_gmem {
+	struct guest_memfd *gmem;
+	struct iommufd_vdevice *ivdev;
+};
+
+#endif /* VFIO_PCI_GMEM_H */
diff --git a/drivers/vfio/pci/vfio_pci_gmem_access.c b/drivers/vfio/pci/vfio_pci_gmem_access.c
new file mode 100644
index 000000000000..40219f05345b
--- /dev/null
+++ b/drivers/vfio/pci/vfio_pci_gmem_access.c
@@ -0,0 +1,101 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Shared/private access and guest revocation are controlled by guest_memfd. */
+#include <linux/guest_memfd.h>
+#include <linux/iommufd.h>
+#include <linux/vfio_pci_core.h>
+
+#include "vfio_pci_gmem.h"
+#include "vfio_pci_priv.h"
+
+/**
+ * vfio_pci_gmem_invalidate() - notify KVM of BAR mappings invalidate
+ * @vdev: VFIO PCI device whose BAR translations are being revoked
+ *
+ * Called by vfio_pci_zap_bars() after it removes host BAR mappings. Notify
+ * guest_memfd synchronously so KVM removes the corresponding shared guest
+ * mappings. This helper does not remove host mappings or change attributes.
+ * Skip notification if no provider is attached.
+ *
+ */
+void vfio_pci_gmem_invalidate(struct vfio_pci_core_device *vdev)
+{
+	struct vfio_pci_gmem *ctx = vdev->gmem;
+
+	lockdep_assert_held_write(&vdev->memory_lock);
+	if (ctx)
+		ctx->gmem->invalidate(ctx->gmem);
+}
+
+/**
+ * vfio_pci_gmem_access_allowed() - check a host BAR access against its range
+ * @vdev: VFIO PCI device supplying the BAR
+ * @offset: byte offset in the VFIO region namespace
+ * @size: access length in bytes
+ *
+ * Consult completed guest_memfd attributes for the requested interval.
+ * Private mappings in another interval do not block this access. Try the
+ * inode lock instead of waiting in the reverse order from conversion.
+ *
+ * Return: true when the requested interval is accessible.
+ */
+bool vfio_pci_gmem_access_allowed(struct vfio_pci_core_device *vdev,
+		u64 offset, u64 size)
+{
+	struct vfio_pci_gmem *ctx = vdev->gmem;
+
+	lockdep_assert_held(&vdev->memory_lock);
+	if (!ctx)
+		return true;
+
+	/* Attachment publishes the context before resolving its vDEVICE. */
+	if (!ctx->ivdev)
+		return false;
+
+	return ctx->gmem->host_accessible(ctx->gmem,
+					  offset, size);
+}
+
+/**
+ * vfio_pci_gmem_io_access_allowed() - resolve a mapped BAR address
+ * @vdev: VFIO PCI device supplying the iomap
+ * @io: address used by ordinary VFIO read, write or ioeventfd
+ * @size: access width in bytes
+ *
+ * Standard BAR iomaps are cached in barmap. Resolve those to file offsets
+ * before applying range admission. Generic VFIO's remaining memory I/O uses
+ * a temporary ROM iomap, which cannot be bound to a device guest_memfd.
+ * Callers supply VFIO's own validated iomap pointers.
+ *
+ * Return: true when the I/O interval is accessible.
+ */
+bool vfio_pci_gmem_io_access_allowed(struct vfio_pci_core_device *vdev,
+		void __iomem *io, size_t size)
+{
+	unsigned long addr = (unsigned long)io;
+	int bar;
+
+	lockdep_assert_held(&vdev->memory_lock);
+	if (!vdev->gmem)
+		return vfio_pci_gmem_access_allowed(vdev, 0, size);
+
+	for (bar = 0; bar < PCI_STD_NUM_BARS; bar++) {
+		unsigned long base = (unsigned long)vdev->barmap[bar];
+		u64 offset;
+
+		if (!base || addr < base)
+			continue;
+
+		offset = addr - base;
+		if (offset >= pci_resource_len(vdev->pdev, bar))
+			continue;
+
+		if (size > pci_resource_len(vdev->pdev, bar) - offset)
+			return false;
+
+		return vfio_pci_gmem_access_allowed(vdev,
+				VFIO_PCI_INDEX_TO_OFFSET(bar) + offset, size);
+	}
+	/* The ROM has no private provider mappings, but lifecycle checks apply. */
+	return vfio_pci_gmem_access_allowed(vdev,
+			VFIO_PCI_INDEX_TO_OFFSET(VFIO_PCI_ROM_REGION_INDEX), size);
+}
diff --git a/drivers/vfio/pci/vfio_pci_rdwr.c b/drivers/vfio/pci/vfio_pci_rdwr.c
index 7f14dd46de17..54c1b3d2f725 100644
--- a/drivers/vfio/pci/vfio_pci_rdwr.c
+++ b/drivers/vfio/pci/vfio_pci_rdwr.c
@@ -44,7 +44,9 @@ int vfio_pci_core_iowrite##size(struct vfio_pci_core_device *vdev,	\
 {									\
 	if (test_mem) {							\
 		down_read(&vdev->memory_lock);				\
-		if (!__vfio_pci_memory_enabled(vdev)) {			\
+		if (!__vfio_pci_memory_enabled(vdev) ||			\
+		    !vfio_pci_gmem_io_access_allowed(vdev, io,		\
+						     sizeof(u##size))) {	\
 			up_read(&vdev->memory_lock);			\
 			return -EIO;					\
 		}							\
@@ -70,7 +72,9 @@ int vfio_pci_core_ioread##size(struct vfio_pci_core_device *vdev,	\
 {									\
 	if (test_mem) {							\
 		down_read(&vdev->memory_lock);				\
-		if (!__vfio_pci_memory_enabled(vdev)) {			\
+		if (!__vfio_pci_memory_enabled(vdev) ||			\
+		    !vfio_pci_gmem_io_access_allowed(vdev, io,		\
+						     sizeof(u##size))) {	\
 			up_read(&vdev->memory_lock);			\
 			return -EIO;					\
 		}							\
@@ -380,7 +384,10 @@ static int vfio_pci_ioeventfd_handler(void *opaque, void *unused)
 	if (ioeventfd->test_mem) {
 		if (!down_read_trylock(&vdev->memory_lock))
 			return 1; /* Lock contended, use thread */
-		if (!__vfio_pci_memory_enabled(vdev)) {
+		if (!__vfio_pci_memory_enabled(vdev) ||
+		    !vfio_pci_gmem_access_allowed(vdev,
+				VFIO_PCI_INDEX_TO_OFFSET(ioeventfd->bar) +
+				ioeventfd->pos, ioeventfd->count)) {
 			up_read(&vdev->memory_lock);
 			return 0;
 		}
diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index dc15b462126e..9f8aa81eb1c9 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -151,8 +151,30 @@ struct vfio_pci_core_device {
 	struct rw_semaphore	memory_lock;
 	atomic_t		mmap_count;
 	struct list_head	dmabufs;
+	/* Attached provider; publication/removal require memory_lock for write. */
+	struct vfio_pci_gmem *gmem;
 };
 
+#if IS_ENABLED(CONFIG_VFIO_PCI_GMEM)
+void vfio_pci_gmem_invalidate(struct vfio_pci_core_device *vdev);
+bool vfio_pci_gmem_access_allowed(struct vfio_pci_core_device *vdev,
+				u64 offset, u64 size);
+bool vfio_pci_gmem_io_access_allowed(struct vfio_pci_core_device *vdev,
+				   void __iomem *io, size_t size);
+#else
+static inline void vfio_pci_gmem_invalidate(struct vfio_pci_core_device *vdev) {}
+static inline bool vfio_pci_gmem_access_allowed(struct vfio_pci_core_device *vdev,
+					     u64 offset, u64 size)
+{
+	return true;
+}
+static inline bool vfio_pci_gmem_io_access_allowed(struct vfio_pci_core_device *vdev,
+						void __iomem *io, size_t size)
+{
+	return true;
+}
+#endif
+
 enum vfio_pci_io_width {
 	VFIO_PCI_IO_WIDTH_1 = 1,
 	VFIO_PCI_IO_WIDTH_2 = 2,
-- 
2.43.0

[RFC PATCH v2 08/13] vfio/pci: Provide guest_memfd backing for PCI BARs

[Aneesh Kumar K.V (Arm)]

Allow an ordinary VFIO PCI device fd to supply BAR memory to
guest_memfd. Validate eligible BAR ranges and provide their PFNs for
shared guest mappings. Keep the VFIO file and its IOMMUFD vDEVICE alive
while the provider is attached.

Reject attachment while VFIO BAR mappings or dma-buf exports exist.

Hold the vDEVICE MMIO mutex across conversion and attribute updates.
Before private conversion, take the VFIO memory lock for writing to wait
for ongoing BAR reads and writes to finish. Host access checks try
inode invalidate lock without waiting, avoiding deadlock with
conversion, which acquires that lock first.

Cc: Alex Williamson <alex@shazbot.org>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 Documentation/virt/kvm/api.rst   |   6 +
 drivers/vfio/pci/Makefile        |   1 +
 drivers/vfio/pci/vfio_pci.c      |   3 +
 drivers/vfio/pci/vfio_pci_gmem.c | 325 +++++++++++++++++++++++++++++++
 drivers/vfio/pci/vfio_pci_gmem.h |   3 +
 include/linux/vfio_pci_core.h    |   1 +
 6 files changed, 339 insertions(+)
 create mode 100644 drivers/vfio/pci/vfio_pci_gmem.c

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index 078f672a115b..0bc38d53a1b3 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -6533,6 +6533,12 @@ With USE_RESOURCE, the resource mapping superblock supplies provider
 operations. Callbacks identify folio or PFN backing. PFN providers require
 INIT_SHARED and a VM supporting protected resource mappings; mmap is disabled.
 
+For a VFIO PCI device resource, creation fails with EBUSY if any VFIO BAR
+VMA or dma-buf export exists. This includes unfaulted mappings and mappings
+inherited by another process. Userspace must remove all such mappings before
+creating the device guest_memfd. While the provider is attached, mmap() on
+any VFIO fd for that device fails with EBUSY.
+
 When the KVM MMU performs a PFN lookup to service a guest fault, the fault will
 always be consumed from guest_memfd, regardless of whether it is a shared or a
 private fault (unless in-place conversion is disabled and the backing
diff --git a/drivers/vfio/pci/Makefile b/drivers/vfio/pci/Makefile
index d491b38ad9c2..1abb7448c8b1 100644
--- a/drivers/vfio/pci/Makefile
+++ b/drivers/vfio/pci/Makefile
@@ -4,6 +4,7 @@ vfio-pci-core-y := vfio_pci_core.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_con
 vfio-pci-core-$(CONFIG_VFIO_PCI_ZDEV_KVM) += vfio_pci_zdev.o
 vfio-pci-core-$(CONFIG_VFIO_PCI_DMABUF) += vfio_pci_dmabuf.o
 vfio-pci-core-$(CONFIG_VFIO_PCI_GMEM) += vfio_pci_gmem_access.o
+vfio-pci-core-$(CONFIG_VFIO_PCI_GMEM) += vfio_pci_gmem.o
 obj-$(CONFIG_VFIO_PCI_CORE) += vfio-pci-core.o
 
 vfio-pci-y := vfio_pci.o
diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c
index 830369ff878d..ad4e612c3336 100644
--- a/drivers/vfio/pci/vfio_pci.c
+++ b/drivers/vfio/pci/vfio_pci.c
@@ -147,6 +147,9 @@ static int vfio_pci_init_dev(struct vfio_device *core_vdev)
 }
 
 static const struct vfio_device_ops vfio_pci_ops = {
+#if IS_ENABLED(CONFIG_VFIO_PCI_GMEM)
+	.gmem_provider_ops = &vfio_pci_gmem_provider_ops,
+#endif
 	.name		= "vfio-pci",
 	.init		= vfio_pci_init_dev,
 	.release	= vfio_pci_core_release_dev,
diff --git a/drivers/vfio/pci/vfio_pci_gmem.c b/drivers/vfio/pci/vfio_pci_gmem.c
new file mode 100644
index 000000000000..8cb5f0cb3bb8
--- /dev/null
+++ b/drivers/vfio/pci/vfio_pci_gmem.c
@@ -0,0 +1,325 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include <linux/guest_memfd.h>
+#include <linux/iommufd.h>
+#include <linux/overflow.h>
+#include <linux/pci.h>
+#include <linux/slab.h>
+#include <linux/vfio_pci_core.h>
+
+#include "../vfio.h"
+#include "vfio_pci_priv.h"
+#include "vfio_pci_gmem.h"
+
+/**
+ * vfio_gmem_resolve_bar_range() - validate and resolve a VFIO BAR interval
+ * @ctx: attached VFIO provider
+ * @offset: byte offset in the VFIO region namespace
+ * @size: interval length in bytes
+ * @bar: output BAR index
+ * @bar_offset: output byte offset within that BAR
+ * @pa: output starting physical address
+ *
+ * Check alignment, mmap eligibility, and wrapper and BAR bounds. These checks
+ * establish the provider interval; the VMM approves the protected REC
+ * physical-address claim.
+ * Output parameters are valid only on success.
+ *
+ * Return: 0 on success or -EINVAL for an ineligible interval.
+ */
+static int vfio_gmem_resolve_bar_range(struct vfio_pci_gmem *ctx, u64 offset,
+		u64 size, u32 *bar, u64 *bar_offset, phys_addr_t *pa)
+{
+	struct pci_dev *pdev = ctx->vdev->pdev;
+	resource_size_t start, len;
+	u32 index = offset >> VFIO_PCI_OFFSET_SHIFT;
+	u64 off = offset & ((1ULL << VFIO_PCI_OFFSET_SHIFT) - 1);
+
+	if (!size || !PAGE_ALIGNED(offset | size))
+		return -EINVAL;
+
+	if (offset >= ctx->gmem->size ||
+	    size > ctx->gmem->size - offset)
+		return -EINVAL;
+
+	if (index >= PCI_STD_NUM_BARS)
+		return -EINVAL;
+
+	start = pci_resource_start(pdev, index);
+	len = pci_resource_len(pdev, index);
+
+	if (!ctx->vdev->bar_mmap_supported[index] ||
+	    !(pci_resource_flags(pdev, index) & IORESOURCE_MEM) ||
+	    !PAGE_ALIGNED(start))
+		return -EINVAL;
+
+	if (off >= len || size > len - off)
+		return -EINVAL;
+
+	/* The interval must not cross into the next VFIO region. */
+	if (size > (1ULL << VFIO_PCI_OFFSET_SHIFT) - off)
+		return -EINVAL;
+
+	if (check_add_overflow(start, off, pa))
+		return -EINVAL;
+
+	*bar = index;
+	*bar_offset = off;
+	return 0;
+}
+
+/**
+ * vfio_gmem_has_private_mmio() - check for private BAR ranges
+ * @data: attached VFIO provider context
+ *
+ * Return: true if the core has committed private BAR ranges.
+ */
+static bool vfio_gmem_has_private_mmio(void *data)
+{
+	struct vfio_pci_gmem *ctx = data;
+
+	lockdep_assert_held(&ctx->ivdev->mmio_lock);
+	return ctx->gmem->has_private(ctx->gmem);
+}
+
+/**
+ * vfio_gmem_attach() - attach the ordinary VFIO PCI fd to guest_memfd
+ * @file: open VFIO device file supplying the BAR namespace
+ * @gdev: guest_memfd core callbacks and owning VM
+ *
+ * Require an opened IOMMUFD-backed file and exclude another provider and
+ * dma-buf exports. Pin the source file and the existing vDEVICE after its
+ * backend validates the VM. VFIO BAR VMAs and provider attachment are mutually
+ * exclusive for the entire provider lifetime.
+ *
+ * Return: Provider context on success, or an ERR_PTR() on failure.
+ */
+static void *vfio_gmem_attach(struct file *file, struct guest_memfd *gdev)
+{
+	struct vfio_device_file *df = file->private_data;
+	struct vfio_pci_core_device *vdev =
+		container_of(df->device, struct vfio_pci_core_device, vdev);
+	struct vfio_pci_gmem *ctx;
+	struct iommufd_vdevice *ivdev;
+	int ret;
+
+	ctx = kzalloc_obj(*ctx);
+	if (!ctx)
+		return ERR_PTR(-ENOMEM);
+
+	ctx->vdev = vdev;
+	ctx->gmem = gdev;
+
+	mutex_lock(&vdev->vdev.dev_set->lock);
+	/* Paired with smp_store_release() after vfio_df_open(). */
+	if (!smp_load_acquire(&df->access_granted) || !df->iommufd) {
+		ret = -EINVAL;
+		goto fail;
+	}
+
+	down_write(&vdev->memory_lock);
+	/*
+	 * Allow only one guest_memfd provider per device. Existing dma-buf
+	 * exports bypass guest_memfd access checks. A BAR VMA can fault even
+	 * if it has never installed a PTE. Exclude both under memory_lock,
+	 * paired with mmap admission and dma-buf export.
+	 */
+	if (vdev->gmem || atomic_read(&vdev->mmap_count) ||
+	    !list_empty(&vdev->dmabufs)) {
+		up_write(&vdev->memory_lock);
+		ret = -EBUSY;
+		goto fail;
+	}
+	vdev->gmem = ctx;
+	up_write(&vdev->memory_lock);
+	ivdev = iommufd_device_attach_mmio_provider(vdev->vdev.iommufd_device,
+						    gdev->kvm, ctx,
+						    vfio_gmem_has_private_mmio,
+						    &ctx->owner);
+	if (IS_ERR(ivdev)) {
+		ret = PTR_ERR(ivdev);
+		scoped_guard(rwsem_write, &vdev->memory_lock)
+			vdev->gmem = NULL;
+		goto fail;
+	}
+	scoped_guard(rwsem_write, &vdev->memory_lock)
+		ctx->ivdev = ivdev;
+	/* Defer VFIO last-close and IOMMUFD unbind until gmem is released. */
+	ctx->resource = get_file(file);
+	mutex_unlock(&vdev->vdev.dev_set->lock);
+	return ctx;
+fail:
+	mutex_unlock(&vdev->vdev.dev_set->lock);
+	kfree(ctx);
+	return ERR_PTR(ret);
+}
+
+/**
+ * vfio_gmem_bind() - bind an eligible BAR interval to a memslot
+ * @data: provider context returned by vfio_gmem_attach()
+ * @offset: byte offset in the VFIO region namespace
+ * @size: length in bytes
+ * @binding: receives the pinned mapping owner and virtual device ID
+ *
+ * Validate the BAR interval and reserve the BAR through the ordinary
+ * VFIO iomap machinery.
+ *
+ * Called with the inode invalidate lock held;
+ * Return: 0 on success or a negative error code.
+ */
+static int vfio_gmem_bind(void *data, u64 offset, u64 size,
+		struct guest_memfd_device_binding *binding)
+{
+	struct vfio_pci_gmem *ctx = data;
+	phys_addr_t pa;
+	u64 off;
+	u32 bar;
+	int ret;
+
+	guard(iommufd_vdevice_mmio)(ctx->ivdev);
+	ret = vfio_gmem_resolve_bar_range(ctx, offset, size, &bar, &off, &pa);
+	if (ret)
+		return ret;
+
+	/* Reserve the physical resource with the ordinary VFIO BAR machinery. */
+	ret = PTR_ERR_OR_ZERO(vfio_pci_core_get_iomap(ctx->vdev, bar));
+	if (!ret) {
+		binding->owner = ctx->owner;
+		binding->vdev_id = ctx->ivdev->virt_id;
+	}
+	return ret;
+}
+
+/**
+ * vfio_gmem_get_pfn() - obtain a shared BAR PFN for a guest fault
+ * @data: attached provider context
+ * @offset: page-aligned byte offset in the VFIO region namespace
+ * @pfn: receives the bare BAR PFN
+ *
+ * Check shared-access admission, power state and PCI memory decoding under
+ * the VFIO memory lock before resolving the resource. The PFN has no RAM
+ * page reference to release. The core checks completed attributes and the
+ * fault path checks the invalidation sequence before installing a mapping.
+ *
+ * Guest_memfd calls this only for shared device ranges. Private mappings use
+ * the physical address from an approved protected device request through
+ * kvm_arch_gmem_device_map(). Installing private mappings through ordinary
+ * PFN faults would bypass that approval path.
+ *
+ * Called with the inode invalidate lock held
+ * Return: 0 on success, -EAGAIN while shared access is unavailable, -ENODEV
+ * after detachment, or a resource-validation error.
+ */
+static int vfio_gmem_get_pfn(void *data, u64 offset, unsigned long *pfn)
+{
+	struct vfio_pci_gmem *ctx = data;
+	struct vfio_pci_core_device *vdev = ctx->vdev;
+	phys_addr_t pa;
+	u64 off;
+	u32 bar;
+	int ret;
+
+	guard(rwsem_read)(&vdev->memory_lock);
+	if (vdev->pm_runtime_engaged || !__vfio_pci_memory_enabled(vdev))
+		return -EAGAIN;
+
+	ret = vfio_gmem_resolve_bar_range(ctx, offset, PAGE_SIZE,
+					  &bar, &off, &pa);
+	if (!ret)
+		*pfn = PHYS_PFN(pa);
+	return ret;
+}
+
+/**
+ * vfio_gmem_prepare_conversion() - serialize device range conversion
+ * @context: attached provider context
+ * @req: protected private request, or NULL for shared cleanup
+ *
+ * Check the virtual device ID before private mapping. The core has already
+ * invalidated shared guest mappings; VFIO userspace mappings are excluded.
+ * Keep that mutex held across architecture cleanup and attribute publication.
+ *
+ * Called with the inode invalidate lock held. On success,
+ * vfio_gmem_finish_conversion() must release the retained MMIO mutex.
+ * Return: 0 with the MMIO mutex held, or a negative error with it released.
+ */
+static int vfio_gmem_prepare_conversion(void *context,
+		const struct guest_memfd_device_request *req)
+{
+	struct vfio_pci_gmem *ctx = context;
+
+	iommufd_vdevice_mmio_lock(ctx->ivdev);
+	if (req && req->vdev_id != ctx->ivdev->virt_id) {
+		iommufd_vdevice_mmio_unlock(ctx->ivdev);
+		return -EPERM;
+	}
+	if (req) {
+		/*
+		 * VFIO BAR read/write and ioeventfd accesses hold memory_lock for
+		 * reading through the MMIO operation. Wait for accesses that passed
+		 * the shared-range check before conversion took the inode invalidate
+		 * lock. New accesses to the attached range are denied because
+		 * host_accessible() cannot acquire that lock for reading with
+		 * filemap_invalidate_trylock_shared().
+		 */
+		down_write(&ctx->vdev->memory_lock);
+		up_write(&ctx->vdev->memory_lock);
+	}
+	return 0; /* Keep mmio_lock held until finish_conversion(). */
+}
+
+/**
+ * vfio_gmem_finish_conversion() - release the conversion mutex
+ * @context: attached provider context
+ * @req: protected private request, or NULL for shared cleanup
+ *
+ * Release the MMIO mutex after the core publishes completed attributes.
+ * Subsequent faults and host I/O check their own range; no device-wide private
+ * or shared admission state is published here.
+ *
+ * Called with the inode invalidate lock and the vDEVICE MMIO mutex held
+ * after successful preparation.
+ */
+static void vfio_gmem_finish_conversion(void *context,
+		const struct guest_memfd_device_request *req)
+{
+	struct vfio_pci_gmem *ctx = context;
+
+	iommufd_vdevice_mmio_unlock(ctx->ivdev);
+}
+
+/**
+ * vfio_gmem_release() - release a provider after successful core cleanup
+ * @data: provider context to release
+ *
+ * Detach from the pinned vDEVICE, clear the VFIO provider association and
+ * drop the source file reference before freeing the context.
+ *
+ * Takes the VFIO device-set lock and the vDEVICE MMIO and VFIO memory
+ * locks while detaching.
+ */
+static void vfio_gmem_release(void *data)
+{
+	struct vfio_pci_gmem *ctx = data;
+	struct vfio_pci_core_device *vdev = ctx->vdev;
+
+	mutex_lock(&vdev->vdev.dev_set->lock);
+	iommufd_vdevice_detach_mmio_provider(ctx->ivdev);
+	down_write(&vdev->memory_lock);
+	vdev->gmem = NULL;
+	up_write(&vdev->memory_lock);
+	mutex_unlock(&vdev->vdev.dev_set->lock);
+	fput(ctx->resource);
+	kfree(ctx);
+}
+
+const struct guest_memfd_provider_operations vfio_pci_gmem_provider_ops = {
+	.attach = vfio_gmem_attach,
+	.bind = vfio_gmem_bind,
+	.get_pfn = vfio_gmem_get_pfn,
+	.prepare_conversion = vfio_gmem_prepare_conversion,
+	.finish_conversion = vfio_gmem_finish_conversion,
+	.release = vfio_gmem_release,
+};
+EXPORT_SYMBOL_GPL(vfio_pci_gmem_provider_ops);
+
+MODULE_IMPORT_NS("IOMMUFD");
diff --git a/drivers/vfio/pci/vfio_pci_gmem.h b/drivers/vfio/pci/vfio_pci_gmem.h
index a8edd20357e7..b4ca831196fd 100644
--- a/drivers/vfio/pci/vfio_pci_gmem.h
+++ b/drivers/vfio/pci/vfio_pci_gmem.h
@@ -9,7 +9,10 @@ struct vfio_pci_core_device;
 
 struct vfio_pci_gmem {
 	struct guest_memfd *gmem;
+	struct vfio_pci_core_device *vdev;
 	struct iommufd_vdevice *ivdev;
+	struct file *resource;
+	unsigned long owner;
 };
 
 #endif /* VFIO_PCI_GMEM_H */
diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index 9f8aa81eb1c9..716b1dc443be 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -156,6 +156,7 @@ struct vfio_pci_core_device {
 };
 
 #if IS_ENABLED(CONFIG_VFIO_PCI_GMEM)
+extern const struct guest_memfd_provider_operations vfio_pci_gmem_provider_ops;
 void vfio_pci_gmem_invalidate(struct vfio_pci_core_device *vdev);
 bool vfio_pci_gmem_access_allowed(struct vfio_pci_core_device *vdev,
 				u64 offset, u64 size);
-- 
2.43.0

[RFC PATCH v2 09/13] KVM: guest_memfd: Add a provider detach callback

[Aneesh Kumar K.V (Arm)]

Add detach() to the guest_memfd provider operations. Providers must disable
new bindings and private requests and stop and drain callbacks that access
the VM before returning.

This separates quiescing VM access from release(), which may run after
the guest_memfd file and its VM reference have been released.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 include/linux/guest_memfd.h | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/include/linux/guest_memfd.h b/include/linux/guest_memfd.h
index f0c62889f823..cfc26294d7ea 100644
--- a/include/linux/guest_memfd.h
+++ b/include/linux/guest_memfd.h
@@ -47,6 +47,8 @@ struct guest_memfd {
  * @get_pfn: Resolve a shared resource offset without taking a folio reference.
  * @prepare_conversion: Prepare protected PFN conversion; NULL request shares.
  * @finish_conversion: Finish conversion after attribute publication.
+ * @detach: Disable new bindings and private requests before final cleanup.
+ *          Stop and drain callbacks that access the VM before returning.
  *
  * Provider operations supply exactly one of alloc_folio() and get_pfn().
  * PFN providers start shared and require the bind and conversion callbacks
@@ -65,6 +67,7 @@ struct guest_memfd_provider_operations {
 		const struct guest_memfd_device_request *req);
 	void (*finish_conversion)(void *provider,
 		const struct guest_memfd_device_request *req);
+	void (*detach)(void *provider);
 };
 
 #endif /* _LINUX_GUEST_MEMFD_H */
-- 
2.43.0

[RFC PATCH v2 10/13] vfio/pci: Detach guest_memfd access before teardown

[Aneesh Kumar K.V (Arm)]

Implement the guest_memfd detach callback and forward it through the VFIO
filesystem provider. Require the backing driver to supply the callback.

Mark the provider detached under the vDEVICE MMIO mutex and VFIO memory
lock, draining in-flight access and invalidation callbacks. Reject further
host accesses, PFN lookups, bindings and private conversions while allowing
shared cleanup.

Cc: Alex Williamson <alex@shazbot.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 drivers/vfio/pci/vfio_pci_gmem.c        | 25 +++++++++++++++++++++++++
 drivers/vfio/pci/vfio_pci_gmem.h        |  6 ++++++
 drivers/vfio/pci/vfio_pci_gmem_access.c |  7 +++++--
 drivers/vfio/vfio_main.c                | 10 ++++++++--
 4 files changed, 44 insertions(+), 4 deletions(-)

diff --git a/drivers/vfio/pci/vfio_pci_gmem.c b/drivers/vfio/pci/vfio_pci_gmem.c
index 8cb5f0cb3bb8..1ab65e43eb1d 100644
--- a/drivers/vfio/pci/vfio_pci_gmem.c
+++ b/drivers/vfio/pci/vfio_pci_gmem.c
@@ -176,6 +176,8 @@ static int vfio_gmem_bind(void *data, u64 offset, u64 size,
 	int ret;
 
 	guard(iommufd_vdevice_mmio)(ctx->ivdev);
+	if (ctx->callbacks_detached)
+		return -ENODEV;
 	ret = vfio_gmem_resolve_bar_range(ctx, offset, size, &bar, &off, &pa);
 	if (ret)
 		return ret;
@@ -219,6 +221,8 @@ static int vfio_gmem_get_pfn(void *data, u64 offset, unsigned long *pfn)
 	int ret;
 
 	guard(rwsem_read)(&vdev->memory_lock);
+	if (ctx->callbacks_detached)
+		return -ENODEV;
 	if (vdev->pm_runtime_engaged || !__vfio_pci_memory_enabled(vdev))
 		return -EAGAIN;
 
@@ -248,6 +252,10 @@ static int vfio_gmem_prepare_conversion(void *context,
 	struct vfio_pci_gmem *ctx = context;
 
 	iommufd_vdevice_mmio_lock(ctx->ivdev);
+	if (req && ctx->callbacks_detached) {
+		iommufd_vdevice_mmio_unlock(ctx->ivdev);
+		return -ENODEV;
+	}
 	if (req && req->vdev_id != ctx->ivdev->virt_id) {
 		iommufd_vdevice_mmio_unlock(ctx->ivdev);
 		return -EPERM;
@@ -287,6 +295,22 @@ static void vfio_gmem_finish_conversion(void *context,
 	iommufd_vdevice_mmio_unlock(ctx->ivdev);
 }
 
+/**
+ * vfio_gmem_detach() - stop invalidation callbacks before core teardown
+ * @data: attached provider context
+ *
+ * Mark the provider detached under the VFIO memory lock. VFIO userspace
+ * mappings remain excluded until release clears the provider association.
+ */
+static void vfio_gmem_detach(void *data)
+{
+	struct vfio_pci_gmem *ctx = data;
+
+	guard(iommufd_vdevice_mmio)(ctx->ivdev);
+	scoped_guard(rwsem_write, &ctx->vdev->memory_lock)
+		ctx->callbacks_detached = true;
+}
+
 /**
  * vfio_gmem_release() - release a provider after successful core cleanup
  * @data: provider context to release
@@ -318,6 +342,7 @@ const struct guest_memfd_provider_operations vfio_pci_gmem_provider_ops = {
 	.get_pfn = vfio_gmem_get_pfn,
 	.prepare_conversion = vfio_gmem_prepare_conversion,
 	.finish_conversion = vfio_gmem_finish_conversion,
+	.detach = vfio_gmem_detach,
 	.release = vfio_gmem_release,
 };
 EXPORT_SYMBOL_GPL(vfio_pci_gmem_provider_ops);
diff --git a/drivers/vfio/pci/vfio_pci_gmem.h b/drivers/vfio/pci/vfio_pci_gmem.h
index b4ca831196fd..76a2fbe1472a 100644
--- a/drivers/vfio/pci/vfio_pci_gmem.h
+++ b/drivers/vfio/pci/vfio_pci_gmem.h
@@ -13,6 +13,12 @@ struct vfio_pci_gmem {
 	struct iommufd_vdevice *ivdev;
 	struct file *resource;
 	unsigned long owner;
+	/*
+	 * Disables callbacks and new bind/private requests. Set once with
+	 * both the vDEVICE MMIO and VFIO memory locks held; readers hold
+	 * either lock. Shared cleanup remains allowed after detachment.
+	 */
+	bool callbacks_detached;
 };
 
 #endif /* VFIO_PCI_GMEM_H */
diff --git a/drivers/vfio/pci/vfio_pci_gmem_access.c b/drivers/vfio/pci/vfio_pci_gmem_access.c
index 40219f05345b..b723a0877ff3 100644
--- a/drivers/vfio/pci/vfio_pci_gmem_access.c
+++ b/drivers/vfio/pci/vfio_pci_gmem_access.c
@@ -14,7 +14,7 @@
  * Called by vfio_pci_zap_bars() after it removes host BAR mappings. Notify
  * guest_memfd synchronously so KVM removes the corresponding shared guest
  * mappings. This helper does not remove host mappings or change attributes.
- * Skip notification if no provider is attached.
+ * Skip notification if no provider is attached or its callbacks are detached.
  *
  */
 void vfio_pci_gmem_invalidate(struct vfio_pci_core_device *vdev)
@@ -22,7 +22,7 @@ void vfio_pci_gmem_invalidate(struct vfio_pci_core_device *vdev)
 	struct vfio_pci_gmem *ctx = vdev->gmem;
 
 	lockdep_assert_held_write(&vdev->memory_lock);
-	if (ctx)
+	if (ctx && !ctx->callbacks_detached)
 		ctx->gmem->invalidate(ctx->gmem);
 }
 
@@ -35,6 +35,7 @@ void vfio_pci_gmem_invalidate(struct vfio_pci_core_device *vdev)
  * Consult completed guest_memfd attributes for the requested interval.
  * Private mappings in another interval do not block this access. Try the
  * inode lock instead of waiting in the reverse order from conversion.
+ * Detachment excludes access during teardown.
  *
  * Return: true when the requested interval is accessible.
  */
@@ -51,6 +52,8 @@ bool vfio_pci_gmem_access_allowed(struct vfio_pci_core_device *vdev,
 	if (!ctx->ivdev)
 		return false;
 
+	if (ctx->callbacks_detached)
+		return false;
 	return ctx->gmem->host_accessible(ctx->gmem,
 					  offset, size);
 }
diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
index 2051e14ac616..d6ea08c08e03 100644
--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -274,9 +274,9 @@ static void *vfio_gmem_provider_attach(struct file *file,
 	if (!ops || !ops->attach || !ops->release)
 		return ERR_PTR(-EOPNOTSUPP);
 
-	/* VFIO requires PFN lookup, binding and conversion. */
+	/* VFIO requires PFN lookup, binding, conversion and detachment. */
 	if (!ops->get_pfn || !ops->bind || !ops->prepare_conversion ||
-	    !ops->finish_conversion)
+	    !ops->finish_conversion || !ops->detach)
 		return ERR_PTR(-EOPNOTSUPP);
 
 	/* VFIO provides PFNs, not folios. */
@@ -340,7 +340,12 @@ static void vfio_gmem_provider_finish_conversion(
 	provider->ops->finish_conversion(provider->data, req);
 }
 
+static void vfio_gmem_provider_detach(void *data)
+{
+	struct vfio_gmem_provider *provider = data;
 
+	provider->ops->detach(provider->data);
+}
 
 static const struct guest_memfd_provider_operations vfio_gmem_provider_ops = {
 	.attach = vfio_gmem_provider_attach,
@@ -349,6 +354,7 @@ static const struct guest_memfd_provider_operations vfio_gmem_provider_ops = {
 	.get_pfn = vfio_gmem_provider_get_pfn,
 	.prepare_conversion = vfio_gmem_provider_prepare_conversion,
 	.finish_conversion = vfio_gmem_provider_finish_conversion,
+	.detach = vfio_gmem_provider_detach,
 };
 
 static const struct super_operations vfio_fs_super_ops = {
-- 
2.43.0

[RFC PATCH v2 11/13] KVM: guest_memfd: Remove device mappings on final close

[Aneesh Kumar K.V (Arm)]

On final guest_memfd close, detach the device provider before removing
private device mappings. Keep memslot bindings available so cleanup can
resolve device ranges to guest addresses.

Hold the inode invalidate lock across detach and conversion cleanup, and
clear the core file pointer before releasing the file-owned VM reference.
Require PFN providers to supply detach() and reject it for folio providers.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 Documentation/virt/kvm/api.rst |  3 ++-
 virt/kvm/guest_memfd.c         | 27 ++++++++++++++++++++++++---
 2 files changed, 26 insertions(+), 4 deletions(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index 0bc38d53a1b3..ca81e2762656 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -6537,7 +6537,8 @@ For a VFIO PCI device resource, creation fails with EBUSY if any VFIO BAR
 VMA or dma-buf export exists. This includes unfaulted mappings and mappings
 inherited by another process. Userspace must remove all such mappings before
 creating the device guest_memfd. While the provider is attached, mmap() on
-any VFIO fd for that device fails with EBUSY.
+any VFIO fd for that device fails with EBUSY. Closing the guest_memfd releases
+the provider once its last reference is dropped and cleanup completes.
 
 When the KVM MMU performs a PFN lookup to service a guest fault, the fault will
 always be consumed from guest_memfd, regardless of whether it is a shared or a
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index d2b1e2e9dfee..6476b64c5928 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -340,6 +340,7 @@ static void kvm_gmem_invalidate_end(struct inode *inode, pgoff_t start,
 static int kvm_gmem_device_convert(struct inode *inode, pgoff_t start,
 		pgoff_t nr_pages, const struct guest_memfd_device_request *req,
 		const struct kvm_memory_slot *slot);
+static void kvm_gmem_device_close(struct inode *inode);
 
 static long kvm_gmem_punch_hole(struct inode *inode, loff_t offset, loff_t len)
 {
@@ -462,6 +463,9 @@ static int kvm_gmem_release(struct inode *inode, struct file *file)
 
 	filemap_invalidate_lock(inode->i_mapping);
 
+	if (gmem_provider_has_pfns(GMEM_I(inode)))
+		kvm_gmem_device_close(inode);
+
 	xa_for_each(&f->bindings, index, slot)
 		WRITE_ONCE(slot->gmem.file, NULL);
 
@@ -1339,6 +1343,23 @@ int kvm_gmem_device_map(struct kvm *kvm, u64 gpa, u64 size, u64 pa, u64 vdev_id)
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_gmem_device_map);
 
+static void kvm_gmem_device_close(struct inode *inode)
+{
+	struct gmem_inode *gi = GMEM_I(inode);
+	int ret;
+
+	gi->provider_ops->detach(gi->provider);
+	ret = gi->provider_ops->prepare_conversion(gi->provider, NULL);
+	if (!ret) {
+		ret = kvm_gmem_device_make_shared(inode, 0,
+						  i_size_read(inode) >> PAGE_SHIFT);
+		if (!ret)
+			WRITE_ONCE(gi->device_has_private, false);
+		gi->provider_ops->finish_conversion(gi->provider, NULL);
+	}
+	WRITE_ONCE(gi->device.gmem_file, NULL);
+}
+
 static int kvm_gmem_validate_provider(struct kvm *kvm,
 		struct gmem_inode *gi,
 		const struct guest_memfd_provider_operations *ops)
@@ -1355,14 +1376,14 @@ static int kvm_gmem_validate_provider(struct kvm *kvm,
 	if (!ops->get_pfn) {
 		/* PFN-specific callbacks are invalid for folio providers. */
 		if (ops->bind || ops->prepare_conversion ||
-		    ops->finish_conversion)
+		    ops->finish_conversion || ops->detach)
 			return -EOPNOTSUPP;
 		return 0;
 	}
 
-	/* PFN providers must support binding and conversion. */
+	/* PFN providers must support binding, conversion and detachment. */
 	if (!ops->bind || !ops->prepare_conversion ||
-	    !ops->finish_conversion)
+	    !ops->finish_conversion || !ops->detach)
 		return -EOPNOTSUPP;
 
 	/* PFN providers do not manage folios. */
-- 
2.43.0

[RFC PATCH v2 12/13] KVM: Invalidate guest_memfd mappings before removing memslot bindings

[Aneesh Kumar K.V (Arm)]

Unmap guest_memfd ranges before deleting the memslot. Remove private
device mappings before committing deletion, and invalidate remaining
mappings while holding the inode invalidate lock.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 virt/kvm/guest_memfd.c | 70 ++++++++++++++++++++++++++++++++++++++++--
 virt/kvm/guest_memfd.h |  6 ++++
 virt/kvm/kvm_main.c    |  4 +++
 3 files changed, 77 insertions(+), 3 deletions(-)

diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 6476b64c5928..05ab01925532 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -1307,6 +1307,58 @@ static int kvm_gmem_device_convert(struct inode *inode, pgoff_t start,
 	return ret;
 }
 
+/**
+ * kvm_gmem_prepare_memslot_delete() - remove guest_memfd mappings before deletion
+ * @slot: original guest_memfd memslot being removed
+ *
+ * The caller has replaced the active slot with an entry marked
+ * KVM_MEMSLOT_INVALID, preventing new guest faults from creating mappings.
+ * Remove private device mappings, then invalidate the remaining guest_memfd
+ * mappings, including shared mappings.
+ *
+ * Perform this cleanup while the guest_memfd binding still provides the
+ * file-offset-to-GPA translation needed to locate the guest mappings.
+ * The device provider remains attached until guest_memfd close.
+ *
+ * Caller holds slots_lock.
+ * Return: 0 on success or a negative error code.
+ */
+int kvm_gmem_prepare_memslot_delete(const struct kvm_memory_slot *slot)
+{
+	struct file *file = READ_ONCE(slot->gmem.file);
+	pgoff_t start = slot->gmem.pgoff;
+	pgoff_t end = start + slot->npages;
+	struct inode *inode;
+	int ret;
+
+	/* slots_lock keeps the bindings alive until file release can finish. */
+	file = get_file_active(&file);
+	if (!file) {
+		/* Let .release drain the bindings before deletion can discard them. */
+		if (READ_ONCE(slot->gmem.file))
+			return -EBUSY;
+		return 0;
+	}
+
+	inode = file_inode(file);
+	if (gmem_provider_has_pfns(GMEM_I(inode))) {
+		ret = kvm_gmem_device_convert(inode, start, slot->npages, NULL, NULL);
+		if (ret)
+			goto out;
+	}
+
+	filemap_invalidate_lock(inode->i_mapping);
+	__kvm_gmem_invalidate_start(file->private_data, start, end,
+				    kvm_gmem_get_all_gfns_filter(inode));
+	__kvm_gmem_invalidate_end(file->private_data, start, end);
+	filemap_invalidate_unlock(inode->i_mapping);
+	ret = 0;
+out:
+	fput(file);
+	return ret;
+}
+
+
 int kvm_gmem_device_map(struct kvm *kvm, u64 gpa, u64 size, u64 pa, u64 vdev_id)
 {
 	struct guest_memfd_device_request req = { gpa, pa, vdev_id };
@@ -1608,11 +1660,19 @@ int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot,
 	return r;
 }
 
-static void __kvm_gmem_unbind(struct kvm_memory_slot *slot, struct gmem_file *f)
+static void __kvm_gmem_unbind(struct kvm_memory_slot *slot, struct file *file)
 {
+	struct gmem_file *f = file->private_data;
 	unsigned long start = slot->gmem.pgoff;
 	unsigned long end = start + slot->npages;
 
+	/*
+	 * Unmap this slot's guest mappings before removing it from f->bindings.
+	 * The invalidation helper walks f->bindings to find the slots to unmap.
+	 */
+	__kvm_gmem_invalidate_start(f, start, end,
+				    kvm_gmem_get_all_gfns_filter(file_inode(file)));
+	__kvm_gmem_invalidate_end(f, start, end);
 	xa_store_range(&f->bindings, start, end - 1, NULL, GFP_KERNEL);
 
 	/*
@@ -1644,12 +1704,16 @@ void kvm_gmem_unbind(struct kvm_memory_slot *slot)
 	 * until the caller drops slots_lock.
 	 */
 	if (!file) {
-		__kvm_gmem_unbind(slot, slot->gmem.file->private_data);
+		struct file *closing_file = slot->gmem.file;
+
+		filemap_invalidate_lock(closing_file->f_mapping);
+		__kvm_gmem_unbind(slot, closing_file);
+		filemap_invalidate_unlock(closing_file->f_mapping);
 		return;
 	}
 
 	filemap_invalidate_lock(file->f_mapping);
-	__kvm_gmem_unbind(slot, file->private_data);
+	__kvm_gmem_unbind(slot, file);
 	filemap_invalidate_unlock(file->f_mapping);
 }
 
diff --git a/virt/kvm/guest_memfd.h b/virt/kvm/guest_memfd.h
index 0f9c6f840838..6fad6adfdadc 100644
--- a/virt/kvm/guest_memfd.h
+++ b/virt/kvm/guest_memfd.h
@@ -11,6 +11,7 @@ int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args);
 int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot,
 		  unsigned int fd, uoff_t offset);
 void kvm_gmem_unbind(struct kvm_memory_slot *slot);
+int kvm_gmem_prepare_memslot_delete(const struct kvm_memory_slot *slot);
 #else
 static inline int kvm_gmem_init(struct module *module)
 {
@@ -29,6 +30,11 @@ static inline void kvm_gmem_unbind(struct kvm_memory_slot *slot)
 {
 	WARN_ON_ONCE(1);
 }
+
+static inline int kvm_gmem_prepare_memslot_delete(const struct kvm_memory_slot *slot)
+{
+	return 0;
+}
 #endif /* CONFIG_KVM_GUEST_MEMFD */
 
 #endif /* __KVM_GUEST_MEMFD_H__ */
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index f3f46a8563b3..58da57b8a268 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -1696,6 +1696,10 @@ static int kvm_prepare_memory_region(struct kvm *kvm,
 	}
 
 	r = kvm_arch_prepare_memory_region(kvm, old, new, change);
+	if (!r && change == KVM_MR_DELETE) {
+		if (old->flags & KVM_MEM_GUEST_MEMFD)
+			r = kvm_gmem_prepare_memslot_delete(old);
+	}
 
 	/* Free the bitmap on failure if it was allocated above. */
 	if (r && new && new->dirty_bitmap && (!old || !old->dirty_bitmap))
-- 
2.43.0

[RFC PATCH v2 13/13] vfio/pci: Allow VFIO BAR mmap with an attached guest_memfd

[Aneesh Kumar K.V (Arm)]

Allow existing and new VFIO BAR VMAs when the attached guest_memfd was
created with GUEST_MEMFD_FLAG_MMAP. Keep BAR VMA exclusion when the flag
is absent. CPU mappings use the VFIO fd; mmap on the PFN-backed
guest_memfd fd remains disabled.

Invalidate existing BAR mappings, including huge mappings, during
attachment. Subsequent faults use base-page mappings so each page's
shared-range attributes are checked separately. Hold memory_lock from
the attribute check through PTE insertion.

Invalidate BAR mappings before private conversion and on provider
detach. Taking memory_lock for writing waits for active
faults, BAR read/write and ioeventfd accesses before invalidation.

Cc: Alex Williamson <alex@shazbot.org>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 Documentation/virt/kvm/api.rst          | 29 +++++++++++-----
 drivers/vfio/pci/vfio_pci_core.c        | 16 ++++++---
 drivers/vfio/pci/vfio_pci_gmem.c        | 46 ++++++++++++++++---------
 drivers/vfio/pci/vfio_pci_gmem.h        |  2 ++
 drivers/vfio/pci/vfio_pci_gmem_access.c | 31 +++++++++++++++++
 include/linux/vfio_pci_core.h           |  5 +++
 6 files changed, 99 insertions(+), 30 deletions(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index ca81e2762656..c1a660d56ba0 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -6514,8 +6514,9 @@ The capability KVM_CAP_GUEST_MEMFD_FLAGS enumerates the `flags` that can be
 specified via KVM_CREATE_GUEST_MEMFD.  Currently defined flags:
 
   ============================= ================================================
-  GUEST_MEMFD_FLAG_MMAP         Enable using mmap() on the guest_memfd file
-                                descriptor.
+  GUEST_MEMFD_FLAG_MMAP         Enable userspace mappings. For VFIO PCI
+                                resources, mappings use the resource fd;
+                                otherwise they use the guest_memfd fd.
   GUEST_MEMFD_FLAG_INIT_SHARED  Make all memory in the file shared during
                                 KVM_CREATE_GUEST_MEMFD (memory files created
                                 without INIT_SHARED will be marked private).
@@ -6531,13 +6532,23 @@ specified via KVM_CREATE_GUEST_MEMFD.  Currently defined flags:
 
 With USE_RESOURCE, the resource mapping superblock supplies provider
 operations. Callbacks identify folio or PFN backing. PFN providers require
-INIT_SHARED and a VM supporting protected resource mappings; mmap is disabled.
-
-For a VFIO PCI device resource, creation fails with EBUSY if any VFIO BAR
-VMA or dma-buf export exists. This includes unfaulted mappings and mappings
-inherited by another process. Userspace must remove all such mappings before
-creating the device guest_memfd. While the provider is attached, mmap() on
-any VFIO fd for that device fails with EBUSY. Closing the guest_memfd releases
+INIT_SHARED and a VM supporting protected resource mappings. Mapping a
+PFN-backed guest_memfd fd is not supported.
+
+For a VFIO PCI device resource, creation fails with EBUSY if another provider
+or any dma-buf export exists. Without GUEST_MEMFD_FLAG_MMAP, creation also fails
+with EBUSY if any VFIO BAR VMA exists, including unfaulted or inherited VMAs.
+While attached without MMAP, mmap() on any VFIO fd for the device fails with
+EBUSY. With MMAP, existing BAR VMAs and new VFIO mmap() calls are allowed.
+Attachment revokes existing BAR translations, including huge mappings. While
+attached, faults use base-page mappings and check the guest_memfd attributes
+for the faulting page.
+
+Conversion to private memory revokes VFIO BAR translations before delegation.
+Access to a private page raises SIGBUS; conversion back to shared permits new
+faults on the existing VMA. Reset, power and memory-decode changes also revoke
+BAR translations. Detachment revokes translations and blocks faults until
+provider release clears the association. Closing the guest_memfd releases
 the provider once its last reference is dropped and cleanup completes.
 
 When the KVM MMU performs a PFN lookup to service a guest fault, the fault will
diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index 1258343a6832..96ec45a23505 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -1713,9 +1713,7 @@ static void vfio_pci_zap_bars(struct vfio_pci_core_device *vdev)
 	loff_t end = VFIO_PCI_INDEX_TO_OFFSET(VFIO_PCI_ROM_REGION_INDEX);
 	loff_t len = end - start;
 
-	/* An attached provider excludes every VFIO BAR VMA. */
-	if (!vdev->gmem)
-		unmap_mapping_range(core_vdev->inode->i_mapping, start, len, true);
+	unmap_mapping_range(core_vdev->inode->i_mapping, start, len, true);
 	vfio_pci_gmem_invalidate(vdev);
 }
 
@@ -1766,6 +1764,15 @@ vm_fault_t vfio_pci_vmf_insert_pfn(struct vfio_pci_core_device *vdev,
 	if (vdev->pm_runtime_engaged || !__vfio_pci_memory_enabled(vdev))
 		return VM_FAULT_SIGBUS;
 
+	/* Check each page separately while guest_memfd controls BAR access. */
+	if (order && vdev->gmem)
+		return VM_FAULT_FALLBACK;
+
+	/* memory_lock keeps revocation from racing admission and PTE insertion. */
+	if (!vfio_pci_gmem_access_allowed(vdev,
+				(u64)vmf->pgoff << PAGE_SHIFT, PAGE_SIZE))
+		return VM_FAULT_SIGBUS;
+
 	if (!order)
 		return vmf_insert_pfn(vmf->vma, vmf->address, pfn);
 
@@ -1842,7 +1849,8 @@ int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct vm_area_struct *vma
 	void __iomem *bar_io;
 
 	guard(rwsem_read)(&vdev->memory_lock);
-	if (vdev->gmem)
+
+	if (!vfio_pci_gmem_mmap_allowed(vdev))
 		return -EBUSY;
 
 	index = vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT);
diff --git a/drivers/vfio/pci/vfio_pci_gmem.c b/drivers/vfio/pci/vfio_pci_gmem.c
index 1ab65e43eb1d..e4e1b0de3cf6 100644
--- a/drivers/vfio/pci/vfio_pci_gmem.c
+++ b/drivers/vfio/pci/vfio_pci_gmem.c
@@ -1,6 +1,7 @@
 // SPDX-License-Identifier: GPL-2.0-only
 #include <linux/guest_memfd.h>
 #include <linux/iommufd.h>
+#include <linux/kvm.h>
 #include <linux/overflow.h>
 #include <linux/pci.h>
 #include <linux/slab.h>
@@ -88,8 +89,9 @@ static bool vfio_gmem_has_private_mmio(void *data)
  *
  * Require an opened IOMMUFD-backed file and exclude another provider and
  * dma-buf exports. Pin the source file and the existing vDEVICE after its
- * backend validates the VM. VFIO BAR VMAs and provider attachment are mutually
- * exclusive for the entire provider lifetime.
+ * backend validates the VM. Without MMAP, exclude BAR VMAs for the provider
+ * lifetime. With MMAP, revoke existing BAR mappings before attachment
+ * completes; subsequent faults check guest_memfd attributes.
  *
  * Return: Provider context on success, or an ERR_PTR() on failure.
  */
@@ -119,18 +121,28 @@ static void *vfio_gmem_attach(struct file *file, struct guest_memfd *gdev)
 	down_write(&vdev->memory_lock);
 	/*
 	 * Allow only one guest_memfd provider per device. Existing dma-buf
-	 * exports bypass guest_memfd access checks. A BAR VMA can fault even
-	 * if it has never installed a PTE. Exclude both under memory_lock,
-	 * paired with mmap admission and dma-buf export.
+	 * exports bypass guest_memfd access checks and cannot be revoked by
+	 * private conversion. BAR VMAs require MMAP: attachment revokes their
+	 * PTEs and subsequent faults check the provider's range attributes.
 	 */
-	if (vdev->gmem || atomic_read(&vdev->mmap_count) ||
-	    !list_empty(&vdev->dmabufs)) {
+	if (vdev->gmem || !list_empty(&vdev->dmabufs)) {
 		up_write(&vdev->memory_lock);
 		ret = -EBUSY;
 		goto fail;
 	}
+
+	/* Without MMAP, exclude unfaulted and inherited BAR VMAs too. */
+	if (!(gdev->flags & GUEST_MEMFD_FLAG_MMAP) &&
+	    atomic_read(&vdev->mmap_count)) {
+		up_write(&vdev->memory_lock);
+		ret = -EBUSY;
+		goto fail;
+	}
+
 	vdev->gmem = ctx;
 	up_write(&vdev->memory_lock);
+	/* Faults deny access until the vDEVICE and guest_memfd file are ready. */
+	vfio_gmem_invalidate_mmio(ctx);
 	ivdev = iommufd_device_attach_mmio_provider(vdev->vdev.iommufd_device,
 						    gdev->kvm, ctx,
 						    vfio_gmem_has_private_mmio,
@@ -238,8 +250,8 @@ static int vfio_gmem_get_pfn(void *data, u64 offset, unsigned long *pfn)
  * @context: attached provider context
  * @req: protected private request, or NULL for shared cleanup
  *
- * Check the virtual device ID before private mapping. The core has already
- * invalidated shared guest mappings; VFIO userspace mappings are excluded.
+ * Check the virtual device ID and revoke VFIO userspace mappings before
+ * private mapping. The core has already invalidated shared guest mappings.
  * Keep that mutex held across architecture cleanup and attribute publication.
  *
  * Called with the inode invalidate lock held. On success,
@@ -262,15 +274,14 @@ static int vfio_gmem_prepare_conversion(void *context,
 	}
 	if (req) {
 		/*
-		 * VFIO BAR read/write and ioeventfd accesses hold memory_lock for
-		 * reading through the MMIO operation. Wait for accesses that passed
-		 * the shared-range check before conversion took the inode invalidate
-		 * lock. New accesses to the attached range are denied because
+		 * VFIO faults, BAR read/write and ioeventfd accesses hold memory_lock
+		 * for reading through PTE insertion or the MMIO operation. Drain
+		 * admitted accesses and revoke existing PTEs before delegation.
+		 * New accesses to the attached range are denied because
 		 * host_accessible() cannot acquire that lock for reading with
 		 * filemap_invalidate_trylock_shared().
 		 */
-		down_write(&ctx->vdev->memory_lock);
-		up_write(&ctx->vdev->memory_lock);
+		vfio_gmem_invalidate_mmio(ctx);
 	}
 	return 0; /* Keep mmio_lock held until finish_conversion(). */
 }
@@ -299,8 +310,8 @@ static void vfio_gmem_finish_conversion(void *context,
  * vfio_gmem_detach() - stop invalidation callbacks before core teardown
  * @data: attached provider context
  *
- * Mark the provider detached under the VFIO memory lock. VFIO userspace
- * mappings remain excluded until release clears the provider association.
+ * Mark the provider detached under the VFIO memory lock and revoke BAR
+ * translations. Faults remain blocked until release clears the association.
  */
 static void vfio_gmem_detach(void *data)
 {
@@ -309,6 +320,7 @@ static void vfio_gmem_detach(void *data)
 	guard(iommufd_vdevice_mmio)(ctx->ivdev);
 	scoped_guard(rwsem_write, &ctx->vdev->memory_lock)
 		ctx->callbacks_detached = true;
+	vfio_gmem_invalidate_mmio(ctx);
 }
 
 /**
diff --git a/drivers/vfio/pci/vfio_pci_gmem.h b/drivers/vfio/pci/vfio_pci_gmem.h
index 76a2fbe1472a..91d3894bf8a9 100644
--- a/drivers/vfio/pci/vfio_pci_gmem.h
+++ b/drivers/vfio/pci/vfio_pci_gmem.h
@@ -21,4 +21,6 @@ struct vfio_pci_gmem {
 	bool callbacks_detached;
 };
 
+void vfio_gmem_invalidate_mmio(struct vfio_pci_gmem *ctx);
+
 #endif /* VFIO_PCI_GMEM_H */
diff --git a/drivers/vfio/pci/vfio_pci_gmem_access.c b/drivers/vfio/pci/vfio_pci_gmem_access.c
index b723a0877ff3..b7d15b2c1372 100644
--- a/drivers/vfio/pci/vfio_pci_gmem_access.c
+++ b/drivers/vfio/pci/vfio_pci_gmem_access.c
@@ -2,6 +2,7 @@
 /* Shared/private access and guest revocation are controlled by guest_memfd. */
 #include <linux/guest_memfd.h>
 #include <linux/iommufd.h>
+#include <linux/kvm.h>
 #include <linux/vfio_pci_core.h>
 
 #include "vfio_pci_gmem.h"
@@ -26,6 +27,36 @@ void vfio_pci_gmem_invalidate(struct vfio_pci_core_device *vdev)
 		ctx->gmem->invalidate(ctx->gmem);
 }
 
+/**
+ * vfio_gmem_invalidate_mmio() - revoke VFIO and shared guest BAR mappings
+ * @ctx: attached provider context
+ *
+ * Drain VFIO faults and host I/O, then remove existing BAR PTEs before
+ * delegation or teardown. The BAR zap also invalidates shared guest mappings
+ * while core callbacks remain attached. During private conversion the inode
+ * invalidate lock excludes new access through the host_accessible() check.
+ *
+ * Caller holds the vDEVICE MMIO mutex, except during attachment before the
+ * vDEVICE is assigned. No private conversion can start at that point.
+ */
+void vfio_gmem_invalidate_mmio(struct vfio_pci_gmem *ctx)
+{
+	if (ctx->ivdev)
+		lockdep_assert_held(&ctx->ivdev->mmio_lock);
+
+	vfio_pci_zap_and_down_write_memory_lock(ctx->vdev);
+	up_write(&ctx->vdev->memory_lock);
+}
+
+/* Serialize the creation flag check with attachment under memory_lock. */
+bool vfio_pci_gmem_mmap_allowed(struct vfio_pci_core_device *vdev)
+{
+	struct vfio_pci_gmem *ctx = vdev->gmem;
+
+	lockdep_assert_held(&vdev->memory_lock);
+	return !ctx || (ctx->gmem->flags & GUEST_MEMFD_FLAG_MMAP);
+}
+
 /**
  * vfio_pci_gmem_access_allowed() - check a host BAR access against its range
  * @vdev: VFIO PCI device supplying the BAR
diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index 716b1dc443be..6d2af77295fe 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -158,12 +158,17 @@ struct vfio_pci_core_device {
 #if IS_ENABLED(CONFIG_VFIO_PCI_GMEM)
 extern const struct guest_memfd_provider_operations vfio_pci_gmem_provider_ops;
 void vfio_pci_gmem_invalidate(struct vfio_pci_core_device *vdev);
+bool vfio_pci_gmem_mmap_allowed(struct vfio_pci_core_device *vdev);
 bool vfio_pci_gmem_access_allowed(struct vfio_pci_core_device *vdev,
 				u64 offset, u64 size);
 bool vfio_pci_gmem_io_access_allowed(struct vfio_pci_core_device *vdev,
 				   void __iomem *io, size_t size);
 #else
 static inline void vfio_pci_gmem_invalidate(struct vfio_pci_core_device *vdev) {}
+static inline bool vfio_pci_gmem_mmap_allowed(struct vfio_pci_core_device *vdev)
+{
+	return true;
+}
 static inline bool vfio_pci_gmem_access_allowed(struct vfio_pci_core_device *vdev,
 					     u64 offset, u64 size)
 {
-- 
2.43.0