[RFC PATCH v4 00/25] KVM/VFIO: Add guest_memfd support for device MMIO

[RFC PATCH v4 00/25] KVM/VFIO: Add guest_memfd support for device MMIO

[Aneesh Kumar K.V (Arm)]

Confidential device assignment needs guest_memfd to manage device MMIO as
well as RAM. Device resources such as PCI BARs supply PFNs directly, and
their protected mappings need architecture-specific map and unmap
operations.

This series adds the common guest_memfd framework for device memory and
a VFIO provider for PCI MMIO BARs, with IOMMUFD vDEVICE integration. It
builds on the resource-provider API [1] and the existing folio and RAM PFN
support.

Backing operations separate folio, RAM PFN and device MMIO handling while
sharing attribute updates and invalidation. Device conversion derives GPA
from memslot bindings and PA from provider PFNs. Revocation and cleanup
remove shared and private mappings without reclaiming MMIO PFNs.

Creating guest_memfd from an IOMMUFD-bound VFIO device rejects existing
VFIO BAR mappings, dma-buf exports and ioeventfd handlers, since these
allow access outside guest_memfd. Direct VFIO BAR access is then blocked;
GUEST_MEMFD_FLAG_MMAP enables shared userspace access through guest_memfd.

The device must be in D0. A runtime-PM reference keeps it active, and VFIO
blocks reset, power-state changes and userspace changes to Memory Enable
or Bus Master Enable while it supplies BAR PFNs to guest_memfd. MSI-X
SET_IRQS remains available; userspace must keep the table and PBA pages
shared when using host-managed MSI-X.

The series and the Arm CCA implementation are available together at:

  https://git.gitlab.arm.com/linux-arm/linux-cca.git cca/topics/7.4/tdisp-host

This series depends on:
[1] Resource-provider groundwork:
https://lore.kernel.org/all/20261022-gmem-alternative-providers-v2-0-5dd1dc6c660a@google.com
[2] IOMMUFD/TSM vIOMMU and vDEVICE support:
https://lore.kernel.org/all/20261008055955.4014342-1-aneesh.kumar@kernel.org/

Notes:
* Codex assisted with implementation, review and series organization.

Changes from v3:
https://lore.kernel.org/all/20261101071928.2462549-1-aneesh.kumar@kernel.org
* Include the VFIO PCI provider and IOMMUFD vDEVICE integration alongside
  the common guest_memfd framework; v3 posted only the common framework.
* Replace make_private()/make_shared() with an optional
  convert_attributes() callback, and prepare_private_pfn() with
  prepare_pfn(). Pass attributes to both so backing implementations can
  handle shared and private memory at attribute-update or PFN-lookup time.
* Pass the device context directly to provider attach(), removing
  kvm_gmem_get_device() and avoiding a VFIO dependency on KVM symbols.
* Store has_private_ranges in guest_memfd_device so providers can read
  the core-maintained summary directly instead of calling has_private().
* Serialize VFIO file operations with provider setup and release using
  gmem_lock. Reject existing BAR mappings, dma-buf exports and ioeventfd
  handlers, and block direct VFIO BAR access while guest_memfd uses the
  device. Shared userspace mappings go through guest_memfd.
* Require D0 and hold a runtime-PM reference while guest_memfd uses the
  device. Block VFIO reset, power-state changes and userspace changes to
  Memory Enable or Bus Master Enable during that time.

Changes from v2:
https://lore.kernel.org/all/20261017073149.1303738-1-aneesh.kumar@kernel.org
* Rebase on the v2 alternative-provider series and separate the common
  guest_memfd framework from the VFIO, IOMMUFD and Arm CCA integration.
* Introduce backing operations to separate folio, RAM PFN and device MMIO
  handling while sharing locking, invalidation and attribute updates.
* Replace prepare_conversion()/finish_conversion() with non-failing
  lock()/unlock() callbacks.
* Convert device memory through ordinary file-offset attribute updates.
  Derive GPA from memslot bindings and PA from provider PFNs, removing the
  device request object.
* Allow shared and private device PFN lookup in the common layer. Include
  both mapping types in generic invalidation and leave architecture
  mapping policy to the backend.
* Support device range revocation through the common revoke path without
  reclaiming MMIO PFNs or changing attributes.
* Add shared device mmap through the existing get_pfn() fault path. Keep
  VMA protections and flags in the core
* Require only get_pfn() for device providers; lookup_pfn_range() remains
  required for RAM PFN providers.

Changes from v1:
https://lore.kernel.org/all/20261010072504.536230-1-aneesh.kumar@kernel.org
* Require GUEST_MEMFD_FLAG_MMAP to allow VFIO BAR mmap while a provider is
  attached; disable it by default.
* Use the common guest_memfd_provider_operations for folio and PFN
  providers. Select operations through the resource mapping superblock
  and forward VFIO operations to the device driver.

Cc: Ackerley Tng <ackerleytng@google.com>
Cc: Alex Williamson <alex@shazbot.org>
Cc: David Woodhouse <dwmw2@infradead.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: "Joerg Roedel (AMD)" <joro@8bytes.org>
Cc: Kevin Tian <kevin.tian@intel.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Robin Murphy <robin.murphy@arm.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Alexey Kardashevskiy <aik@amd.com>
Cc: Xu Yilun <yilun.xu@linux.intel.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
Cc: Steven Price <steven.price@arm.com>
Cc: Fred Griffoul <griffoul@gmail.com>
Cc: iommu@lists.linux.dev
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev

Aneesh Kumar K.V (Arm) (25):
  KVM: guest_memfd: Let providers prepare memory regions
  KVM: guest_memfd: Dispatch conversion through backing operations
  KVM: guest_memfd: Let providers coordinate mapping changes
  KVM: guest_memfd: Dispatch revocation through backing operations
  KVM: guest_memfd: Split population by backing type
  KVM: guest_memfd: Support device-backed PFN providers
  KVM: guest_memfd: Support private/shared conversion of device memory
  KVM: guest_memfd: Track private device ranges after attribute updates
  KVM: guest_memfd: Support range revocation of device mappings
  KVM: guest_memfd: Add shared device mmap through PFN providers
  KVM: guest_memfd: Clear private device state on final close
  KVM: guest_memfd: Invalidate mappings before memslot unbind
  vfio: Track device use by guest_memfd
  vfio/pci: Add helpers to check and serialize BAR state changes
  vfio/pci: Check command enable changes before updating device state
  vfio/pci: Check whether BAR state changes are allowed before reset
  vfio/pci: Check low-power requests before updating device state
  vfio/pci: Track userspace BAR mappings
  vfio/pci: Restrict host BAR access for guest_memfd providers
  vfio: Dispatch guest_memfd provider operations through the device file
  vfio: Serialize guest_memfd attachment with device file operations
  vfio/pci: Require D0 before providing guest_memfd BAR backing
  iommufd: Support MMIO provider attachment to vDEVICEs
  vfio/pci: Add a guest_memfd resource provider for MMIO BARs
  KVM: Document device-backed guest_memfd and VFIO usage

 Documentation/virt/kvm/api.rst     |  63 ++-
 Documentation/virt/kvm/locking.rst |  27 ++
 drivers/iommu/iommufd/viommu.c     |  86 ++++
 drivers/vfio/pci/Kconfig           |  11 +
 drivers/vfio/pci/Makefile          |   1 +
 drivers/vfio/pci/vfio_pci.c        |   3 +
 drivers/vfio/pci/vfio_pci_config.c | 134 ++++--
 drivers/vfio/pci/vfio_pci_core.c   | 120 ++++-
 drivers/vfio/pci/vfio_pci_dmabuf.c |   4 +
 drivers/vfio/pci/vfio_pci_gmem.c   | 327 +++++++++++++
 drivers/vfio/pci/vfio_pci_gmem.h   |  17 +
 drivers/vfio/pci/vfio_pci_priv.h   |  28 +-
 drivers/vfio/pci/vfio_pci_rdwr.c   |   4 +
 drivers/vfio/vfio_main.c           | 135 ++++++
 include/linux/guest_memfd.h        |  74 ++-
 include/linux/iommufd.h            |  26 ++
 include/linux/kvm_host.h           |   7 +
 include/linux/vfio.h               |  13 +
 include/linux/vfio_pci_core.h      |   7 +
 mm/shmem.c                         |   3 +-
 samples/kvm/mem_provider_sample.c  |   3 +-
 virt/kvm/guest_memfd.c             | 722 +++++++++++++++++++++++------
 22 files changed, 1627 insertions(+), 188 deletions(-)
 create mode 100644 drivers/vfio/pci/vfio_pci_gmem.c
 create mode 100644 drivers/vfio/pci/vfio_pci_gmem.h


base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
prerequisite-patch-id: 1d4a2fa9dc0f4b560e55f419d23835f0a9fbe485
prerequisite-patch-id: 21acd717af7d0f19db31afe0871a62b3b82ff21b
prerequisite-patch-id: 1d895ab7ebe362ca9b1941de8a58bc0a9a57124a
prerequisite-patch-id: b9003ee26d6e855f8e95aa6978372f009d98e737
prerequisite-patch-id: d8fdf7f17485c6a2905e689a5cd72c76d0d4c446
prerequisite-patch-id: 5fb6db464c1936ed74d2fc2fe4083aa9a3a968c7
prerequisite-patch-id: 03d76de408efeb5a2205079bb61face09255ca60
prerequisite-patch-id: a1635ac11bebba53e057e38c032b1e6b4cefffc4
prerequisite-patch-id: 851431bcda2a44f9475af58c86353be9aa9bbb4e
prerequisite-patch-id: 446c5c5033f7acc066ca98503af62798dca09d3c
prerequisite-patch-id: acbc4e19d554317c64bc813901434005a563da75
prerequisite-patch-id: 2c81e480010f916f7cf823b8db04e3dc90951b14
prerequisite-patch-id: 5b1537647264994178adb5e7f56d50c6cc49b234
prerequisite-patch-id: fa549035ee7387b3fe85177ae4c78b4a725a80b8
prerequisite-patch-id: 9dccd5e814d553649ee13c3f6c1cab4dc652acab
prerequisite-patch-id: 09de20b5a192aacbb036d35eecce3c94cc5680c5
prerequisite-patch-id: d0932c65d2a453e84f7917da27f9f6ae1da95856
prerequisite-patch-id: 7be8f39fdcfe7e504a9ac85e835a5dfb2ee414ba
prerequisite-patch-id: 5ee7e97d4814e41f362261deb9054485e561fc3b
prerequisite-patch-id: 45cd3951b114a26eac94390727287c1221862b5c
prerequisite-patch-id: 6d978b6f0561697e5b145e4b82d79ac8c89f9f31
prerequisite-patch-id: 9bd6979aa47dd00c87a3aa8feb1798486107ac4b
prerequisite-patch-id: 98438bd35d53340738a20fd8724b038c9de1207e
prerequisite-patch-id: 146dfaa64e29a4930adb60e9524c7613f888a241
prerequisite-patch-id: 170e27bc053fede8d24accca69809d3687faf668
prerequisite-patch-id: 789fe5bfb1183f6b9a9b0d817e5d1050a7999e1a
prerequisite-patch-id: 5dd6878d0476a9dd925aa8d7ef1cf18900cbcd82
prerequisite-patch-id: 1f0d1dca2a51df2283f1d83ed57dd9ce4e11b025
prerequisite-patch-id: 503eb8fb0bf9af163f344c2693d374b6a3093fd1
prerequisite-patch-id: 94e4777a58b357458bdce680bbb41c04c0c661f4
prerequisite-patch-id: 4c1f6b548994d239e5093ee1d268ef5a12a76873
prerequisite-patch-id: f6d13ac583d54fe1ae22adde87f5c68d8e8e41ce
prerequisite-patch-id: 7846b44d34507a9af6ac8a3fb263ee446ea56a1d
prerequisite-patch-id: 2d309198e27a3922b5cf5805ac5660ea92029219
prerequisite-patch-id: e1f75965298956ad87bb78969461eff31419b6eb
prerequisite-patch-id: 40ef669df2a64e5b28127059e4cea288d426f6f6
prerequisite-patch-id: d8adb8f100cb13423db02e7e7a834fa241ff04fe
prerequisite-patch-id: 75591ca138e7de26db6cd3e242a7f0ce56c4752c
prerequisite-patch-id: b1e2c46b473783f37490c29140498a0bb35ec7d3
prerequisite-patch-id: bc8b607986a132eef4daa5e20979d3f038d046ba
prerequisite-patch-id: c9de5cd8a95f8102597b49dcf70b8ec5825b9ec7
prerequisite-patch-id: 954606132449080c4e173eb06d114504f87d0322
prerequisite-patch-id: 82a44a46cf9d4cc3fcdc03c362fc4e86d1fc533a
prerequisite-patch-id: 23dbc9968c0b2570f40cd6c3b13337078334aa8d
prerequisite-patch-id: 6c6a5ca6d0a1b60158a47ed21f25889dd71dd9b7
prerequisite-patch-id: e4d1ba4119277d7772dbf6e81f748beab4e62856
prerequisite-patch-id: a32d69bdd1fb21b85547fdc026475e2d94ffbc28
prerequisite-patch-id: 2704fe13a294e1f33cfbee6cd27c11ee392a8b78
prerequisite-patch-id: ebb701b7e050327e798c172d897021806f7e6c35
prerequisite-patch-id: 03db7e48c976a019ce478883002670c4b1a33cd9
prerequisite-patch-id: a18608129e63bbbc6e89a4ca237863ba4c506a63
prerequisite-patch-id: a35607a73bd0329fd91a39bd7b9c855edb4b8798
prerequisite-patch-id: 6d57cd16d6c6e71c3da069c1d9eee1c86bdeee8d
prerequisite-patch-id: b64a5a718f540e7daf2dc4f0703422ad2cc52ebe
prerequisite-patch-id: eb9a9e4d7f4b7e439c5687786628f07c406864a4
prerequisite-patch-id: 2fe22fda60b4e4678bcf62d87bfd82181556af98
prerequisite-patch-id: 26211cbacdbc31a62524aa5ad402b0f3933d433e
prerequisite-patch-id: 02455f9ee70487d8b30b6aada5f5b62ff4e5ebe2
prerequisite-patch-id: 113da62f7845c2a9f86d24ed178cf52756917cc0
prerequisite-patch-id: 9ef3e8b12bc70690470035f0c6f351fa49562659
prerequisite-patch-id: 0953a92b86c0790554b64ebda5d6aed038a6cf0e
prerequisite-patch-id: 877e1c79e389936278122aa1efbdf99053b2d6c7
prerequisite-patch-id: 0b04c869d49a4c1ad96ff7b5d8e3b11c4069f23f
prerequisite-patch-id: 942b7480ddb281b9df74705f1a3a6e23a2440a43
prerequisite-patch-id: 98c3864233a7ba6d789ad260f4af170a020d2929
prerequisite-patch-id: 40dc0f2b1ab5e59f1f5de0be5134101ca896d13d
-- 
2.43.0

[RFC PATCH v4 01/25] KVM: guest_memfd: Let providers prepare memory regions

[Aneesh Kumar K.V (Arm)]

Memory providers may need to check memslot flags and backing ranges
before KVM installs a memslot. Add an optional prepare_memory_region()
callback to the common provider operations

Pass the byte offset, size and requested dirty-logging and read-only
flags, and return any error before publishing the slot. Providers do not
need access to KVM's memslot structure or internal flags. Existing
providers that do not implement the callback keep their current
behavior.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 include/linux/guest_memfd.h |  8 ++++++++
 virt/kvm/guest_memfd.c      | 15 ++++++++++++++-
 2 files changed, 22 insertions(+), 1 deletion(-)

diff --git a/include/linux/guest_memfd.h b/include/linux/guest_memfd.h
index 3e2c0714ca33..17407504307b 100644
--- a/include/linux/guest_memfd.h
+++ b/include/linux/guest_memfd.h
@@ -14,6 +14,12 @@ struct mempolicy;
  * @attach: Attach to a resource file, perform filesystem-specific validation,
  *          and return an opaque provider context.
  * @release: Release provider context and unpin any resources.
+ * @prepare_memory_region: Optional check of a memory range and preparation of
+ *                         access to its backing memory. Called after common
+ *                         file checks and before publishing the memslot.
+ *                         Offset and size are in bytes. The flags contain only
+ *                         KVM_MEM_LOG_DIRTY_PAGES and KVM_MEM_READONLY bits.
+ *                         Return 0 on success or a negative error code.
  * @alloc_folio: Allocate an uninserted folio for a given index and NUMA policy.
  *               Must return a pointer to a struct folio on success, or an
  *               ERR_PTR() error code on failure. Must never return NULL.
@@ -44,6 +50,8 @@ struct mempolicy;
 struct guest_memfd_provider_operations {
 	void *(*attach)(struct inode *gmem_inode, struct file *resource_file);
 	void (*release)(void *provider);
+	int (*prepare_memory_region)(void *provider, u64 offset, u64 size,
+				     u32 flags);
 	struct folio *(*alloc_folio)(void *provider, pgoff_t index,
 				     struct mempolicy *mpol);
 	void (*invalidate_folio)(void *provider, struct folio *folio);
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 48d14a20fc0b..291305888311 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -1367,10 +1367,11 @@ int kvm_gmem_prepare_memory_region(struct kvm *kvm, struct kvm_memory_slot *slot
 				   unsigned int fd, uoff_t offset)
 {
 	uoff_t size = slot->npages << PAGE_SHIFT;
+	struct gmem_inode *gi;
 	struct gmem_file *f;
 	struct inode *inode;
 	struct file *file;
-
+	int ret;
 
 	BUILD_BUG_ON(sizeof(gpa_t) != sizeof(offset));
 	BUILD_BUG_ON(sizeof(gfn_t) != sizeof(slot->gmem.pgoff));
@@ -1390,10 +1391,22 @@ int kvm_gmem_prepare_memory_region(struct kvm *kvm, struct kvm_memory_slot *slot
 		goto err;
 
 	inode = file_inode(file);
+	gi = GMEM_I(inode);
 
 	if (!PAGE_ALIGNED(offset) || offset + size > i_size_read(inode))
 		goto err;
 
+	if (gi->provider_ops->prepare_memory_region) {
+		u32 flags = slot->flags & (KVM_MEM_LOG_DIRTY_PAGES | KVM_MEM_READONLY);
+
+		ret = gi->provider_ops->prepare_memory_region(gi->provider, offset,
+							      size, flags);
+		if (ret) {
+			fput(file);
+			return ret;
+		}
+	}
+
 	/*
 	 * memslots of flag KVM_MEM_GUEST_MEMFD are immutable to change, so
 	 * kvm_gmem_bind() must occur on a new memslot.  Because the memslot
-- 
2.43.0

[RFC PATCH v4 02/25] KVM: guest_memfd: Dispatch conversion through backing operations

[Aneesh Kumar K.V (Arm)]

Device backing needs different validation and architecture conversion
from RAM, while sharing the same locking, guest invalidation and
attribute publication sequence.

Introduce backing operations for prepare_attributes(),
convert_attributes() and prepare_pfn(), selected after provider
attachment. Keep validation separate from range conversion before
attribute publication and preparation after guest PFN lookup. Pass the
target or current attributes to the conversion operations so each
backing can choose which directions require work at each phase. Either
conversion operation can be omitted when no work is needed at that
phase.

Move the existing folio and RAM PFN handling behind these operations.
RAM converts existing backing to shared when attributes change and
prepares private PFNs on lookup using the existing architecture hooks.
Shared conversion remains eager because host faults bypass the guest PFN
preparation operation. Document this ordering at the common conversion
helper, which also permits backends to install private mappings before
the guest resumes.

This provides the common sequence and backing-specific hooks needed by
the subsequent device-memory support.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 virt/kvm/guest_memfd.c | 255 ++++++++++++++++++++++++++++++-----------
 1 file changed, 191 insertions(+), 64 deletions(-)

diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 291305888311..3cb1523ad74e 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -31,6 +31,36 @@ struct gmem_file {
 	struct list_head entry;
 };
 
+struct kvm_gmem_backing_ops {
+	/*
+	 * Validate the transition before guest invalidation and conversion.
+	 * Caller holds inode->i_mapping->invalidate_lock for write. May revoke
+	 * host mappings and check outstanding references. On failure, return an
+	 * error and set err_index to the failing offset; it defaults to the range
+	 * start.
+	 */
+	int (*prepare_attributes)(struct inode *inode, pgoff_t start,
+			size_t nr_pages, u64 attrs, pgoff_t *err_index);
+	/*
+	 * Complete range conversion after guest invalidation starts and before
+	 * attributes are published. Caller holds inode->i_mapping->invalidate_lock
+	 * for write. Shared memory must be safe for host access when the
+	 * attributes become visible; host faults bypass prepare_pfn().
+	 * NULL means no additional conversion is needed at this phase.
+	 */
+	int (*convert_attributes)(struct inode *inode, pgoff_t start, pgoff_t end,
+				  u64 attrs);
+	/*
+	 * Prepare resolved PFNs for their current attributes before returning
+	 * them. Caller holds inode->i_mapping->invalidate_lock for read and the
+	 * folio lock if a folio is present. Must tolerate repeated and concurrent
+	 * lookups, and propagate preparation failures without changing attributes.
+	 * NULL means no work is needed at lookup time.
+	 */
+	int (*prepare_pfn)(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+			   kvm_pfn_t nr_pages, u64 attrs);
+};
+
 struct gmem_inode {
 	struct shared_policy policy;
 	struct inode vfs_inode;
@@ -38,6 +68,7 @@ struct gmem_inode {
 
 	void *provider;
 	const struct guest_memfd_provider_operations *provider_ops;
+	const struct kvm_gmem_backing_ops *backing_ops;
 
 	u64 flags;
 	/*
@@ -49,6 +80,9 @@ struct gmem_inode {
 	struct maple_tree attributes;
 };
 
+static const struct kvm_gmem_backing_ops *
+kvm_gmem_select_backing_ops(struct gmem_inode *gi);
+
 static __always_inline struct gmem_inode *GMEM_I(struct inode *inode)
 {
 	return container_of(inode, struct gmem_inode, vfs_inode);
@@ -101,6 +135,7 @@ static bool gmem_provider_ops_valid(struct gmem_inode *gi,
 
 static int kvm_gmem_attach_resource(struct inode *inode, int resource_fd)
 {
+	struct gmem_inode *gi = GMEM_I(inode);
 	const struct guest_memfd_provider_operations *ops = NULL;
 	void *provider = ERR_PTR(-EOPNOTSUPP);
 	struct file *resource_file = NULL;
@@ -116,7 +151,7 @@ static int kvm_gmem_attach_resource(struct inode *inode, int resource_fd)
 			ops = resource_file->f_op->gmem_provider_ops;
 	}
 
-	if (gmem_provider_ops_valid(GMEM_I(inode), ops))
+	if (gmem_provider_ops_valid(gi, ops))
 		provider = ops->attach(inode, resource_file);
 
 	if (resource_file)
@@ -125,8 +160,9 @@ static int kvm_gmem_attach_resource(struct inode *inode, int resource_fd)
 	if (IS_ERR(provider))
 		return PTR_ERR(provider);
 
-	GMEM_I(inode)->provider = provider;
-	GMEM_I(inode)->provider_ops = ops;
+	gi->provider = provider;
+	gi->provider_ops = ops;
+	gi->backing_ops = kvm_gmem_select_backing_ops(gi);
 
 	return 0;
 }
@@ -928,31 +964,15 @@ static bool kvm_gmem_has_outstanding_references(struct inode *inode,
 }
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
-static void kvm_gmem_make_shared(struct inode *inode, pgoff_t start, pgoff_t end)
+static int kvm_gmem_folio_convert_attributes(struct inode *inode,
+					     pgoff_t start, pgoff_t end, u64 attrs)
 {
-	struct gmem_inode *gi = GMEM_I(inode);
 	struct folio_batch fbatch;
 	pgoff_t next = start;
 	int i;
 
-	if (kvm_gmem_is_pfn_mode(gi)) {
-		pgoff_t index;
-
-		for (index = start; index < end; ) {
-			size_t step = 1;
-			size_t nr_pages;
-			kvm_pfn_t pfn;
-
-			if (!gmem_provider_lookup_pfn_range(gi, index, &pfn, &nr_pages)) {
-				step = min_t(size_t, nr_pages, end - index);
-				kvm_arch_gmem_make_shared(pfn, step);
-			}
-
-			index += step;
-			cond_resched();
-		}
-		return;
-	}
+	if ((attrs & KVM_MEMORY_ATTRIBUTE_PRIVATE) || !kvm_arch_has_gmem_convert())
+		return 0;
 
 	folio_batch_init(&fbatch);
 	while (filemap_get_folios(inode->i_mapping, &next, end - 1, &fbatch)) {
@@ -979,11 +999,127 @@ static void kvm_gmem_make_shared(struct inode *inode, pgoff_t start, pgoff_t end
 		folio_batch_release(&fbatch);
 		cond_resched();
 	}
+
+	return 0;
+}
+
+static int kvm_gmem_ram_pfn_convert_attributes(struct inode *inode,
+					       pgoff_t start, pgoff_t end, u64 attrs)
+{
+	struct gmem_inode *gi = GMEM_I(inode);
+	pgoff_t index;
+
+	if ((attrs & KVM_MEMORY_ATTRIBUTE_PRIVATE) || !kvm_arch_has_gmem_convert())
+		return 0;
+
+	for (index = start; index < end; ) {
+		size_t step = 1;
+		size_t nr_pages;
+		kvm_pfn_t pfn;
+
+		if (!gmem_provider_lookup_pfn_range(gi, index, &pfn, &nr_pages)) {
+			step = min_t(size_t, nr_pages, end - index);
+			kvm_arch_gmem_make_shared(pfn, step);
+		}
+
+		index += step;
+		cond_resched();
+	}
+
+	return 0;
 }
 #else
-static void kvm_gmem_make_shared(struct inode *inode, pgoff_t start, pgoff_t end) {}
+static int kvm_gmem_folio_convert_attributes(struct inode *inode,
+					     pgoff_t start, pgoff_t end, u64 attrs)
+{
+	return 0;
+}
+
+static int kvm_gmem_ram_pfn_convert_attributes(struct inode *inode,
+					       pgoff_t start, pgoff_t end, u64 attrs)
+{
+	return 0;
+}
 #endif
 
+static int kvm_gmem_folio_prepare_attributes(struct inode *inode, pgoff_t start,
+		size_t nr_pages, u64 attrs, pgoff_t *err_index)
+{
+	if (!(attrs & KVM_MEMORY_ATTRIBUTE_PRIVATE))
+		return 0;
+
+	/*
+	 * Forcefully unmap the pages from all userspace page tables,
+	 * and then verify there are no outstanding references, e.g.
+	 * acquired via GUP or similar.  Tell userspace to try again if
+	 * there are outstanding references and hope that whatever has
+	 * pinned the page will put its reference "soon".
+	 */
+	unmap_mapping_pages(inode->i_mapping, start, nr_pages, false);
+	if (kvm_gmem_has_outstanding_references(inode, start, nr_pages, err_index))
+		return -EAGAIN;
+
+	return 0;
+}
+
+static int kvm_gmem_ram_pfn_prepare_attributes(struct inode *inode, pgoff_t start,
+		size_t nr_pages, u64 attrs, pgoff_t *err_index)
+{
+	if (!(attrs & KVM_MEMORY_ATTRIBUTE_PRIVATE))
+		return 0;
+
+	unmap_mapping_pages(inode->i_mapping, start, nr_pages, false);
+	return 0;
+}
+
+static int kvm_gmem_ram_prepare_pfn(struct kvm *kvm, gfn_t gfn,
+				    kvm_pfn_t pfn, kvm_pfn_t nr_pages, u64 attrs)
+{
+	if ((attrs & KVM_MEMORY_ATTRIBUTE_PRIVATE) && kvm_arch_has_gmem_convert())
+		return kvm_arch_gmem_make_private(kvm, gfn, pfn, nr_pages);
+
+	return 0;
+}
+
+static const struct kvm_gmem_backing_ops kvm_gmem_folio_backing_ops = {
+	.prepare_attributes = kvm_gmem_folio_prepare_attributes,
+	.convert_attributes = kvm_gmem_folio_convert_attributes,
+	.prepare_pfn = kvm_gmem_ram_prepare_pfn,
+};
+
+static const struct kvm_gmem_backing_ops kvm_gmem_ram_pfn_backing_ops = {
+	.prepare_attributes = kvm_gmem_ram_pfn_prepare_attributes,
+	.convert_attributes = kvm_gmem_ram_pfn_convert_attributes,
+	.prepare_pfn = kvm_gmem_ram_prepare_pfn,
+};
+
+static const struct kvm_gmem_backing_ops *
+kvm_gmem_select_backing_ops(struct gmem_inode *gi)
+{
+	if (kvm_gmem_is_pfn_mode(gi))
+		return &kvm_gmem_ram_pfn_backing_ops;
+
+	return &kvm_gmem_folio_backing_ops;
+}
+
+/*
+ * Complete conversions that cannot wait for a guest fault before publishing
+ * the new attributes. Shared conversion must finish here because host mmap
+ * faults bypass kvm_gmem_get_pfn(). RAM private preparation can wait for the
+ * guest fault path, but device backing may need private mappings before the
+ * guest resumes, e.g. to validate a guest's device-mapping request.
+ */
+static int kvm_gmem_convert_attributes(struct inode *inode, pgoff_t start,
+		pgoff_t end, u64 attrs)
+{
+	const struct kvm_gmem_backing_ops *ops = GMEM_I(inode)->backing_ops;
+
+	if (ops->convert_attributes)
+		return ops->convert_attributes(inode, start, end, attrs);
+
+	return 0;
+}
+
 static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 				     size_t nr_pages, uint64_t attrs,
 				     pgoff_t *err_index)
@@ -993,57 +1129,39 @@ static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 	struct gmem_inode *gi = GMEM_I(inode);
 	enum kvm_gfn_range_filter filter;
 	pgoff_t end = start + nr_pages;
-	struct maple_tree *mt;
 	struct ma_state mas;
 	int r = 0;
 
-	mt = &gi->attributes;
-
+	*err_index = start;
 	filemap_invalidate_lock(mapping);
 
 	if (kvm_gmem_range_has_attributes(inode, start, nr_pages, attrs))
 		goto out;
 
-	mas_init(&mas, mt, start);
+	mas_init(&mas, &gi->attributes, start);
 	r = kvm_gmem_mas_preallocate(&mas, attrs, start, nr_pages);
-	if (r) {
-		*err_index = start;
+	if (r)
 		goto out;
-	}
-
-	if (to_private) {
-		/*
-		 * Forcefully unmap the pages from all userspace page tables,
-		 * and then verify there are no outstanding references, e.g.
-		 * acquired via GUP or similar.  Tell userspace to try again if
-		 * there are outstanding references and hope that whatever has
-		 * pinned the page will put its reference "soon".
-		 */
-		unmap_mapping_pages(mapping, start, nr_pages, false);
-
-		if (!kvm_gmem_is_pfn_mode(gi) &&
-		    kvm_gmem_has_outstanding_references(inode, start, nr_pages,
-							err_index)) {
-			mas_destroy(&mas);
-			r = -EAGAIN;
-			goto out;
-		}
-	}
 
-	/*
-	 * From this point on guest_memfd has performed necessary
-	 * checks and can proceed to do guest-breaking changes.
-	 */
+	r = gi->backing_ops->prepare_attributes(inode, start, nr_pages, attrs,
+						  err_index);
+	if (r)
+		goto destroy;
 
 	filter = to_private ? KVM_FILTER_SHARED : KVM_FILTER_PRIVATE;
 	kvm_gmem_invalidate_start(inode, start, end, filter);
 
-	if (!to_private && kvm_arch_has_gmem_convert())
-		kvm_gmem_make_shared(inode, start, end);
-
-	mas_store_prealloc(&mas, xa_mk_value(attrs));
+	r = kvm_gmem_convert_attributes(inode, start, end, attrs);
+	if (!r)
+		mas_store_prealloc(&mas, xa_mk_value(attrs));
 
+	if (r)
+		mas_destroy(&mas);
 	kvm_gmem_invalidate_end(inode, start, end);
+	goto out;
+
+destroy:
+	mas_destroy(&mas);
 out:
 	filemap_invalidate_unlock(mapping);
 	return r;
@@ -1551,7 +1669,9 @@ static struct folio *__kvm_gmem_get_pfn(struct file *file,
 int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 		     gfn_t gfn, kvm_pfn_t *pfn, int *max_order)
 {
+	const struct kvm_gmem_backing_ops *ops;
 	pgoff_t index = kvm_gmem_get_index(slot, gfn);
+	struct inode *inode;
 	struct folio *folio;
 	int r = 0, __order;
 
@@ -1561,7 +1681,10 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 	if (!file)
 		return -EFAULT;
 
-	filemap_invalidate_lock_shared(file_inode(file)->i_mapping);
+	inode = file_inode(file);
+	ops = GMEM_I(inode)->backing_ops;
+
+	filemap_invalidate_lock_shared(inode->i_mapping);
 
 	folio = __kvm_gmem_get_pfn(file, slot, index, pfn, max_order);
 	if (IS_ERR(folio)) {
@@ -1569,10 +1692,13 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 		goto out;
 	}
 
-	if (kvm_arch_has_gmem_convert() &&
-	    kvm_gmem_is_private_mem(file_inode(file), index))
-		r = kvm_arch_gmem_make_private(kvm, gfn, *pfn,
-					       (kvm_pfn_t)1 << *max_order);
+	if (ops->prepare_pfn) {
+		u64 attrs = kvm_gmem_is_private_mem(inode, index) ?
+			    KVM_MEMORY_ATTRIBUTE_PRIVATE : 0;
+
+		r = ops->prepare_pfn(kvm, gfn, *pfn,
+				     (kvm_pfn_t)1 << *max_order, attrs);
+	}
 
 	if (folio) {
 		folio_unlock(folio);
@@ -1580,7 +1706,7 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 	}
 
 out:
-	filemap_invalidate_unlock_shared(file_inode(file)->i_mapping);
+	filemap_invalidate_unlock_shared(inode->i_mapping);
 	return r;
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_gmem_get_pfn);
@@ -1722,6 +1848,7 @@ static struct inode *kvm_gmem_alloc_inode(struct super_block *sb)
 	gi->flags = 0;
 	gi->provider = NULL;
 	gi->provider_ops = NULL;
+	gi->backing_ops = NULL;
 	INIT_LIST_HEAD(&gi->gmem_file_list);
 	return &gi->vfs_inode;
 }
-- 
2.43.0

[RFC PATCH v4 03/25] KVM: guest_memfd: Let providers coordinate mapping changes

[Aneesh Kumar K.V (Arm)]

Resource providers may need to serialize host access with guest
invalidation, architecture conversion and publication of new attributes.

Add optional paired lock() and unlock() callbacks without exposing KVM
memslots or architecture requests through the provider interface. Reject
providers that supply only one callback.

Call gmem_provider_lock() with inode->i_mapping->invalidate_lock held
for write, after prepare_attributes() and before guest invalidation
starts. It invokes the provider's optional lock() callback. Call
gmem_provider_unlock() after invalidation ends, regardless of whether
conversion succeeds. Publish attributes before unlocking only when
conversion succeeds.

Subsequent revocation, memslot unbind and final file cleanup use the
same pair without changing attributes.

Existing providers omit both callbacks and retain their current behavior.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 include/linux/guest_memfd.h | 11 +++++++++++
 virt/kvm/guest_memfd.c      | 37 ++++++++++++++++++++++++++++---------
 2 files changed, 39 insertions(+), 9 deletions(-)

diff --git a/include/linux/guest_memfd.h b/include/linux/guest_memfd.h
index 17407504307b..9fc04f1188c1 100644
--- a/include/linux/guest_memfd.h
+++ b/include/linux/guest_memfd.h
@@ -20,6 +20,15 @@ struct mempolicy;
  *                         Offset and size are in bytes. The flags contain only
  *                         KVM_MEM_LOG_DIRTY_PAGES and KVM_MEM_READONLY bits.
  *                         Return 0 on success or a negative error code.
+ * @lock: Optional provider exclusion for mapping and attribute changes. Called
+ *        with the inode invalidate lock held for write, before guest
+ *        invalidation starts. May sleep and must return with exclusion held.
+ *        Supply both lock and unlock, or neither.
+ * @unlock: Release exclusion acquired by lock(). Called after guest
+ *          invalidation ends and any successful attribute publication, with
+ *          the inode invalidate lock still held. The pair also covers range
+ *          revocation, memslot unbind and final file cleanup; those operations
+ *          do not change attributes.
  * @alloc_folio: Allocate an uninserted folio for a given index and NUMA policy.
  *               Must return a pointer to a struct folio on success, or an
  *               ERR_PTR() error code on failure. Must never return NULL.
@@ -52,6 +61,8 @@ struct guest_memfd_provider_operations {
 	void (*release)(void *provider);
 	int (*prepare_memory_region)(void *provider, u64 offset, u64 size,
 				     u32 flags);
+	void (*lock)(void *provider);
+	void (*unlock)(void *provider);
 	struct folio *(*alloc_folio)(void *provider, pgoff_t index,
 				     struct mempolicy *mpol);
 	void (*invalidate_folio)(void *provider, struct folio *folio);
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 3cb1523ad74e..20c72693c210 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -33,19 +33,20 @@ struct gmem_file {
 
 struct kvm_gmem_backing_ops {
 	/*
-	 * Validate the transition before guest invalidation and conversion.
-	 * Caller holds inode->i_mapping->invalidate_lock for write. May revoke
-	 * host mappings and check outstanding references. On failure, return an
-	 * error and set err_index to the failing offset; it defaults to the range
-	 * start.
+	 * Validate the transition before gmem_provider_lock(), guest invalidation
+	 * and conversion. Caller holds inode->i_mapping->invalidate_lock for write.
+	 * May revoke host mappings and check outstanding references. On failure,
+	 * return an error and set err_index to the failing offset; it defaults to
+	 * the range start.
 	 */
 	int (*prepare_attributes)(struct inode *inode, pgoff_t start,
 			size_t nr_pages, u64 attrs, pgoff_t *err_index);
 	/*
 	 * Complete range conversion after guest invalidation starts and before
 	 * attributes are published. Caller holds inode->i_mapping->invalidate_lock
-	 * for write. Shared memory must be safe for host access when the
-	 * attributes become visible; host faults bypass prepare_pfn().
+	 * for write and calls this between gmem_provider_lock() and
+	 * gmem_provider_unlock(). Shared memory must be safe for host access when
+	 * the attributes become visible; host faults bypass prepare_pfn().
 	 * NULL means no additional conversion is needed at this phase.
 	 */
 	int (*convert_attributes)(struct inode *inode, pgoff_t start, pgoff_t end,
@@ -125,6 +126,9 @@ static bool gmem_provider_ops_valid(struct gmem_inode *gi,
 	if (!ops || !ops->attach)
 		return false;
 
+	if (!!ops->lock != !!ops->unlock)
+		return false;
+
 	if (kvm_gmem_is_pfn_mode(gi))
 		return ops->get_pfn && ops->lookup_pfn_range;
 
@@ -226,6 +230,18 @@ static void gmem_provider_put_pfn_range(struct gmem_inode *gi, pgoff_t index,
 		gi->provider_ops->put_pfn_range(gi->provider, index, nr_pages);
 }
 
+static void gmem_provider_lock(struct gmem_inode *gi)
+{
+	if (gi->provider_ops && gi->provider_ops->lock)
+		gi->provider_ops->lock(gi->provider);
+}
+
+static void gmem_provider_unlock(struct gmem_inode *gi)
+{
+	if (gi->provider_ops && gi->provider_ops->unlock)
+		gi->provider_ops->unlock(gi->provider);
+}
+
 static void gmem_provider_release(struct gmem_inode *gi)
 {
 	if (gi->provider_ops && gi->provider_ops->release)
@@ -1121,8 +1137,7 @@ static int kvm_gmem_convert_attributes(struct inode *inode, pgoff_t start,
 }
 
 static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
-				     size_t nr_pages, uint64_t attrs,
-				     pgoff_t *err_index)
+		size_t nr_pages, u64 attrs, pgoff_t *err_index)
 {
 	bool to_private = attrs & KVM_MEMORY_ATTRIBUTE_PRIVATE;
 	struct address_space *mapping = inode->i_mapping;
@@ -1148,6 +1163,9 @@ static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 	if (r)
 		goto destroy;
 
+	/* Call the provider's lock() before invalidation as well as conversion. */
+	gmem_provider_lock(gi);
+
 	filter = to_private ? KVM_FILTER_SHARED : KVM_FILTER_PRIVATE;
 	kvm_gmem_invalidate_start(inode, start, end, filter);
 
@@ -1158,6 +1176,7 @@ static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 	if (r)
 		mas_destroy(&mas);
 	kvm_gmem_invalidate_end(inode, start, end);
+	gmem_provider_unlock(gi);
 	goto out;
 
 destroy:
-- 
2.43.0

[RFC PATCH v4 04/25] KVM: guest_memfd: Dispatch revocation through backing operations

[Aneesh Kumar K.V (Arm)]

Revocation truncates cached folios for folio backing, while RAM PFN
backing unmaps host mappings, reclaims architecture state and optionally
releases provider PFNs. Move that into backing-specific revoke()
operations.

Hold optional provider exclusion across invalidation and revocation,
inside the inode invalidate lock. Keep reclamation and provider PFN
release in the backing operation rather than the common sequence.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 virt/kvm/guest_memfd.c | 62 ++++++++++++++++++++++++++++++------------
 1 file changed, 44 insertions(+), 18 deletions(-)

diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 20c72693c210..4280c3b740ef 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -32,6 +32,8 @@ struct gmem_file {
 };
 
 struct kvm_gmem_backing_ops {
+	int (*revoke)(struct inode *inode, pgoff_t start, pgoff_t end,
+		const unsigned long *pfn, bool put_pfn_range);
 	/*
 	 * Validate the transition before gmem_provider_lock(), guest invalidation
 	 * and conversion. Caller holds inode->i_mapping->invalidate_lock for write.
@@ -473,14 +475,48 @@ static void kvm_gmem_pfn_reclaim(struct gmem_inode *gi, pgoff_t start, pgoff_t e
 static void kvm_gmem_pfn_reclaim(struct gmem_inode *gi, pgoff_t start, pgoff_t end) {}
 #endif
 
+static int kvm_gmem_folio_revoke(struct inode *inode, pgoff_t start, pgoff_t end,
+		const unsigned long *pfn, bool put_pfn_range)
+{
+	loff_t offset = (loff_t)start << PAGE_SHIFT;
+	loff_t len = (loff_t)(end - start) << PAGE_SHIFT;
+
+	truncate_inode_pages_range(inode->i_mapping, offset, offset + len - 1);
+	return 0;
+}
+
+static int kvm_gmem_ram_pfn_revoke(struct inode *inode, pgoff_t start, pgoff_t end,
+		const unsigned long *pfn, bool put_pfn_range)
+{
+	loff_t offset = (loff_t)start << PAGE_SHIFT;
+	loff_t len = (loff_t)(end - start) << PAGE_SHIFT;
+	size_t nr_pages = end - start;
+
+	unmap_mapping_range(inode->i_mapping, offset, len, 0);
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
+	if (pfn)
+		kvm_arch_gmem_reclaim(*pfn, nr_pages);
+	else
+		kvm_gmem_pfn_reclaim(GMEM_I(inode), start, end);
+#endif
+	if (put_pfn_range)
+		gmem_provider_put_pfn_range(GMEM_I(inode), start, nr_pages);
+	return 0;
+}
+
 static int __kvm_gmem_revoke(struct inode *inode, pgoff_t start,
 			      size_t nr_pages, const unsigned long *pfn,
 			      bool put_pfn_range)
 {
 	enum kvm_gfn_range_filter filter = kvm_gmem_get_all_gfns_filter(inode);
 	pgoff_t max_pages = i_size_read(inode) >> PAGE_SHIFT;
-	loff_t offset, len;
+	struct gmem_inode *gi = GMEM_I(inode);
+	const struct kvm_gmem_backing_ops *ops = gi->backing_ops;
 	pgoff_t end;
+	int r;
+
+	if (!ops->revoke)
+		return -EOPNOTSUPP;
 
 	if (start >= max_pages)
 		return 0;
@@ -489,8 +525,6 @@ static int __kvm_gmem_revoke(struct inode *inode, pgoff_t start,
 	if (!nr_pages)
 		return 0;
 
-	offset = (loff_t)start << PAGE_SHIFT;
-	len = (loff_t)nr_pages << PAGE_SHIFT;
 	end = start + nr_pages;
 
 	/*
@@ -499,27 +533,17 @@ static int __kvm_gmem_revoke(struct inode *inode, pgoff_t start,
 	 */
 	filemap_invalidate_lock(inode->i_mapping);
 
+	gmem_provider_lock(gi);
+
 	kvm_gmem_invalidate_start(inode, start, end, filter);
 
-	if (kvm_gmem_is_pfn_mode(GMEM_I(inode))) {
-		unmap_mapping_range(inode->i_mapping, offset, len, 0);
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
-		if (pfn)
-			kvm_arch_gmem_reclaim(*pfn, nr_pages);
-		else
-			kvm_gmem_pfn_reclaim(GMEM_I(inode), start, end);
-#endif
-		if (put_pfn_range)
-			gmem_provider_put_pfn_range(GMEM_I(inode), start, nr_pages);
-	} else {
-		truncate_inode_pages_range(inode->i_mapping, offset, offset + len - 1);
-	}
+	r = ops->revoke(inode, start, end, pfn, put_pfn_range);
 
 	kvm_gmem_invalidate_end(inode, start, end);
-
+	gmem_provider_unlock(gi);
 	filemap_invalidate_unlock(inode->i_mapping);
 
-	return 0;
+	return r;
 }
 
 static long kvm_gmem_punch_hole(struct inode *inode, loff_t offset, loff_t len)
@@ -1098,12 +1122,14 @@ static int kvm_gmem_ram_prepare_pfn(struct kvm *kvm, gfn_t gfn,
 }
 
 static const struct kvm_gmem_backing_ops kvm_gmem_folio_backing_ops = {
+	.revoke = kvm_gmem_folio_revoke,
 	.prepare_attributes = kvm_gmem_folio_prepare_attributes,
 	.convert_attributes = kvm_gmem_folio_convert_attributes,
 	.prepare_pfn = kvm_gmem_ram_prepare_pfn,
 };
 
 static const struct kvm_gmem_backing_ops kvm_gmem_ram_pfn_backing_ops = {
+	.revoke = kvm_gmem_ram_pfn_revoke,
 	.prepare_attributes = kvm_gmem_ram_pfn_prepare_attributes,
 	.convert_attributes = kvm_gmem_ram_pfn_convert_attributes,
 	.prepare_pfn = kvm_gmem_ram_prepare_pfn,
-- 
2.43.0

[RFC PATCH v4 05/25] KVM: guest_memfd: Split population by backing type

[Aneesh Kumar K.V (Arm)]

Preallocation obtains cached folios for folio backing and walks provider
PFN ranges for RAM PFN backing. Both use the same range validation and
shared inode invalidate lock.

Move the two walks into backing-specific populate() operations. Keep
size checks and locking in kvm_gmem_allocate(), preserving signal
handling, folio references and PFN range stepping.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 virt/kvm/guest_memfd.c | 101 +++++++++++++++++++++++++----------------
 1 file changed, 61 insertions(+), 40 deletions(-)

diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 4280c3b740ef..5a4da27f0434 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -32,6 +32,7 @@ struct gmem_file {
 };
 
 struct kvm_gmem_backing_ops {
+	int (*populate)(struct inode *inode, pgoff_t start, pgoff_t end);
 	int (*revoke)(struct inode *inode, pgoff_t start, pgoff_t end,
 		const unsigned long *pfn, bool put_pfn_range);
 	/*
@@ -578,46 +579,11 @@ int kvm_gmem_revoke(struct inode *inode, pgoff_t start, size_t nr_pages,
 }
 EXPORT_SYMBOL_GPL(kvm_gmem_revoke);
 
-static long kvm_gmem_allocate(struct inode *inode, loff_t offset, loff_t len)
+static int kvm_gmem_folio_populate(struct inode *inode,
+		pgoff_t start, pgoff_t end)
 {
-	struct address_space *mapping = inode->i_mapping;
-	struct gmem_inode *gi = GMEM_I(inode);
-	pgoff_t start, index, end;
-	int r;
-
-	/* Dedicated guest is immutable by default. */
-	if (offset + len > i_size_read(inode))
-		return -EINVAL;
-
-	filemap_invalidate_lock_shared(mapping);
-
-	start = offset >> PAGE_SHIFT;
-	end = (offset + len) >> PAGE_SHIFT;
-
-	r = 0;
-	if (kvm_gmem_is_pfn_mode(gi)) {
-		for (index = start; index < end; ) {
-			pgoff_t next_boundary;
-			pgoff_t max_nr_pages;
-			int max_order = 0;
-			kvm_pfn_t pfn;
-
-			if (signal_pending(current)) {
-				r = -EINTR;
-				break;
-			}
-
-			r = gmem_provider_get_pfn(gi, index, &pfn, &max_order);
-			if (r)
-				break;
-
-			max_nr_pages = 1UL << max_order;
-			next_boundary = ALIGN_DOWN(index + max_nr_pages, max_nr_pages);
-			index += min_t(size_t, next_boundary - index, end - index);
-			cond_resched();
-		}
-		goto out_unlock;
-	}
+	pgoff_t index;
+	int r = 0;
 
 	for (index = start; index < end; ) {
 		struct folio *folio;
@@ -645,7 +611,60 @@ static long kvm_gmem_allocate(struct inode *inode, loff_t offset, loff_t len)
 		cond_resched();
 	}
 
-out_unlock:
+	return r;
+}
+
+static int kvm_gmem_ram_pfn_populate(struct inode *inode,
+		pgoff_t start, pgoff_t end)
+{
+	struct gmem_inode *gi = GMEM_I(inode);
+	pgoff_t index;
+	int r = 0;
+
+	for (index = start; index < end; ) {
+		pgoff_t next_boundary;
+		pgoff_t max_nr_pages;
+		int max_order = 0;
+		kvm_pfn_t pfn;
+
+		if (signal_pending(current)) {
+			r = -EINTR;
+			break;
+		}
+
+		r = gmem_provider_get_pfn(gi, index, &pfn, &max_order);
+		if (r)
+			break;
+
+		max_nr_pages = 1UL << max_order;
+		next_boundary = ALIGN_DOWN(index + max_nr_pages, max_nr_pages);
+		index += min_t(size_t, next_boundary - index, end - index);
+		cond_resched();
+	}
+
+	return r;
+}
+
+static long kvm_gmem_allocate(struct inode *inode, loff_t offset, loff_t len)
+{
+	struct address_space *mapping = inode->i_mapping;
+	const struct kvm_gmem_backing_ops *ops = GMEM_I(inode)->backing_ops;
+	pgoff_t start, end;
+	int r;
+
+	if (!ops->populate)
+		return -EOPNOTSUPP;
+
+	/* Dedicated guest is immutable by default. */
+	if (offset + len > i_size_read(inode))
+		return -EINVAL;
+
+	filemap_invalidate_lock_shared(mapping);
+
+	start = offset >> PAGE_SHIFT;
+	end = (offset + len) >> PAGE_SHIFT;
+	r = ops->populate(inode, start, end);
+
 	filemap_invalidate_unlock_shared(mapping);
 
 	return r;
@@ -1122,6 +1141,7 @@ static int kvm_gmem_ram_prepare_pfn(struct kvm *kvm, gfn_t gfn,
 }
 
 static const struct kvm_gmem_backing_ops kvm_gmem_folio_backing_ops = {
+	.populate = kvm_gmem_folio_populate,
 	.revoke = kvm_gmem_folio_revoke,
 	.prepare_attributes = kvm_gmem_folio_prepare_attributes,
 	.convert_attributes = kvm_gmem_folio_convert_attributes,
@@ -1129,6 +1149,7 @@ static const struct kvm_gmem_backing_ops kvm_gmem_folio_backing_ops = {
 };
 
 static const struct kvm_gmem_backing_ops kvm_gmem_ram_pfn_backing_ops = {
+	.populate = kvm_gmem_ram_pfn_populate,
 	.revoke = kvm_gmem_ram_pfn_revoke,
 	.prepare_attributes = kvm_gmem_ram_pfn_prepare_attributes,
 	.convert_attributes = kvm_gmem_ram_pfn_convert_attributes,
-- 
2.43.0

[RFC PATCH v4 06/25] KVM: guest_memfd: Support device-backed PFN providers

[Aneesh Kumar K.V (Arm)]

Device resources such as PCI BARs supply PFNs directly and cannot use
the folio allocation path. Store the mapping owner and device ID in the
core context during attach(), and use them when preparing device
memslots. Keep device memory separate from ordinary PFN-backed RAM.

Require device providers to use PFN_MODE and INIT_SHARED with
architecture support for protected device mappings. Device providers
require get_pfn(), but not the lookup_pfn_range() callback.

Mark device memslots as guest_memfd-only and pass their device binding
to the architecture. Resolve guest PFN lookup through get_pfn() for
either shared or private attributes; the architecture decides how to
establish mappings. Include both mapping types in generic invalidation,
using the memslot binding to select the architecture unmap path. Host
userspace access still requires shared attributes.

Skip RAM population and PFN reclamation for device MMIO, which has no
RAM pages to initialize or reclaim.

Keep mmap, fallocate and attribute changes on device-backed guest_memfd
files disabled until device mapping and conversion support is added.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 Documentation/virt/kvm/api.rst    | 16 ++++--
 include/linux/guest_memfd.h       | 46 +++++++++++++---
 include/linux/kvm_host.h          |  5 ++
 mm/shmem.c                        |  3 +-
 samples/kvm/mem_provider_sample.c |  3 +-
 virt/kvm/guest_memfd.c            | 87 +++++++++++++++++++++++++++----
 6 files changed, 138 insertions(+), 22 deletions(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index bb550e4f11cd..8455a9c7c971 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -6478,8 +6478,10 @@ that refers to it.  guest_memfd files are roughly analogous to files created
 via memfd_create(), e.g. guest_memfd files live in RAM, have volatile storage,
 and are automatically released when the last reference is dropped.  Unlike
 "regular" memfd_create() files, guest_memfd files are bound to their owning
-virtual machine (see below), cannot be mapped, read, or written by userspace,
-and cannot be resized  (guest_memfd files do however support PUNCH_HOLE).
+virtual machine (see below), cannot be read or written by userspace, and
+cannot be resized. Userspace mappings depend on the creation flags and backing
+resource. Ordinary guest_memfd files support PUNCH_HOLE; device-backed files
+reject fallocate().
 
 ::
 
@@ -6528,10 +6530,16 @@ specified via KVM_CREATE_GUEST_MEMFD.  Currently defined flags:
                                 by an external resource provider. When set,
                                 GUEST_MEMFD_FLAG_USE_RESOURCE must also be set,
                                 and the provider must support PFN lookup
-                                (.get_pfn and .lookup_pfn_range) rather than
-                                folio allocation (.alloc_folio).
+                                (.get_pfn and, for RAM providers,
+                                .lookup_pfn_range) rather than folio allocation
+                                (.alloc_folio).
   ============================= ================================================
 
+With USE_RESOURCE, the resource file's operations supply the guest_memfd
+provider. PFN_MODE selects the PFN callbacks; providers that supply device
+operations additionally require INIT_SHARED and a VM supporting protected
+device mappings.
+
 When the KVM MMU performs a PFN lookup to service a guest fault, the fault will
 always be consumed from guest_memfd, regardless of whether it is a shared or a
 private fault (unless in-place conversion is disabled and the backing
diff --git a/include/linux/guest_memfd.h b/include/linux/guest_memfd.h
index 9fc04f1188c1..6bf44c62b7fe 100644
--- a/include/linux/guest_memfd.h
+++ b/include/linux/guest_memfd.h
@@ -8,12 +8,39 @@ struct file;
 struct folio;
 struct inode;
 struct mempolicy;
+struct kvm;
+
+/* Binding data stays valid only while the provider is active. */
+struct guest_memfd_device_binding {
+	/* Architecture-specific mapping owner, e.g. an RMM vDEVICE handle. */
+	unsigned long owner;
+	u64 vdev_id;
+};
+
+/**
+ * struct guest_memfd_device - Device memory context supplied during attach()
+ * @kvm: VM owning the device mappings
+ * @size: File size in bytes
+ * @flags: GUEST_MEMFD_FLAG_* creation flags
+ * @binding: Mapping owner and device ID set by attach(), with the underlying
+ *           resources pinned until release()
+ */
+struct guest_memfd_device {
+	struct kvm *kvm;
+	u64 size;
+	u64 flags;
+	struct guest_memfd_device_binding binding;
+};
 
 /**
  * struct guest_memfd_provider_operations - Operations for external memory providers
  * @attach: Attach to a resource file, perform filesystem-specific validation,
- *          and return an opaque provider context.
- * @release: Release provider context and unpin any resources.
+ *          and return an opaque provider context. Pin resources and callback
+ *          code until release(). Device providers receive their core context
+ *          in the device argument, which is NULL for other providers.
+ * @release: Release provider context and unpin any resources. Must not access
+ *           the VM here; its file reference may be gone.
+ * @device_memory: Backing is device MMIO rather than RAM.
  * @prepare_memory_region: Optional check of a memory range and preparation of
  *                         access to its backing memory. Called after common
  *                         file checks and before publishing the memslot.
@@ -38,8 +65,12 @@ struct mempolicy;
  *           zeroing if necessary. On success, return 0 with the physical frame
  *           number in @pfn and, if @max_order is non-NULL, the maximum page
  *           order of contiguous backing memory containing @pfn. Returns a
- *           negative error code on failure. The provider must define
- *           @lookup_pfn_range when @get_pfn is defined.
+ *           negative error code on failure. RAM PFN providers must also define
+ *           @lookup_pfn_range. Device providers need only @get_pfn and are
+ *           called with the inode invalidate lock held, for read on faults.
+ *           Guest PFN lookup is allowed for either shared or private file
+ *           attributes; the architecture determines how mappings are created.
+ *           Host userspace faults require shared attributes.
  * @lookup_pfn_range: Look up the PFN for a given index. Returns 0 on success
  *                    with the physical frame number in @pfn and the number of
  *                    contiguous pages the mapping is valid for in @nr_pages.
@@ -47,7 +78,8 @@ struct mempolicy;
  *                    is present. If backing memory is not available, return
  *                    -ENOENT. Return a negative error code on other failure.
  *                    The provider must define @get_pfn when @lookup_pfn_range
- *                    is defined.
+ *                    is defined. Not used for device memory, whose protected
+ *                    mappings are removed through architecture device unmap.
  * @put_pfn_range: Release a range of PFNs previously obtained for a given
  *                 index.  The provider does not need to ensure that backing
  *                 memory is present. If backing memory is not available, the
@@ -57,7 +89,8 @@ struct mempolicy;
  * Used by filesystems and device drivers that provide memory for guest_memfd.
  */
 struct guest_memfd_provider_operations {
-	void *(*attach)(struct inode *gmem_inode, struct file *resource_file);
+	void *(*attach)(struct inode *gmem_inode, struct file *resource_file,
+			struct guest_memfd_device *device);
 	void (*release)(void *provider);
 	int (*prepare_memory_region)(void *provider, u64 offset, u64 size,
 				     u32 flags);
@@ -71,6 +104,7 @@ struct guest_memfd_provider_operations {
 	int (*lookup_pfn_range)(void *provider, pgoff_t index,
 				unsigned long *pfn, size_t *nr_pages);
 	void (*put_pfn_range)(void *provider, pgoff_t index, size_t nr_pages);
+	bool device_memory;
 };
 
 #ifdef CONFIG_KVM_GUEST_MEMFD
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 345f97b110fa..f2dd387ab0fa 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -60,6 +60,7 @@
  */
 #define KVM_MEMSLOT_INVALID			(1UL << 16)
 #define KVM_MEMSLOT_GMEM_ONLY			(1UL << 17)
+#define KVM_MEMSLOT_GMEM_DEVICE			(1UL << 18)
 
 /*
  * Bit 63 of the memslot generation number is an "update in-progress flag",
@@ -2613,6 +2614,10 @@ static inline bool kvm_is_private_gfn(struct kvm *kvm, gfn_t gfn)
 #endif /* kvm_arch_has_private_mem */
 
 #ifdef CONFIG_KVM_GUEST_MEMFD
+struct guest_memfd_device_binding;
+bool kvm_arch_gmem_device_supported(struct kvm *kvm);
+void kvm_arch_gmem_device_bind(struct kvm_memory_slot *slot,
+		const struct guest_memfd_device_binding *binding);
 bool kvm_gmem_is_private_gfn(struct kvm *kvm, gfn_t gfn);
 bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm);
 
diff --git a/mm/shmem.c b/mm/shmem.c
index a1c1cc92515b..9f7d71ee46b0 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -5462,7 +5462,8 @@ static const struct inode_operations shmem_special_inode_operations = {
 
 #ifdef CONFIG_KVM_GUEST_MEMFD
 static void *shmem_gmem_attach(struct inode *gmem_inode,
-			       struct file *resource_file)
+			       struct file *resource_file,
+			       struct guest_memfd_device *device)
 {
 	struct inode *inode = file_inode(resource_file);
 	struct shmem_sb_info *sbinfo;
diff --git a/samples/kvm/mem_provider_sample.c b/samples/kvm/mem_provider_sample.c
index 5496e0c7864b..59825203be58 100644
--- a/samples/kvm/mem_provider_sample.c
+++ b/samples/kvm/mem_provider_sample.c
@@ -151,7 +151,8 @@ static long mps_file_ioctl(struct file *file, unsigned int cmd,
 }
 
 static void *mps_gmem_attach(struct inode *gmem_inode,
-			     struct file *resource_file)
+			     struct file *resource_file,
+			     struct guest_memfd_device *device)
 {
 	struct mps_info *info = resource_file->private_data;
 
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 5a4da27f0434..397ead84bdb3 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -74,6 +74,8 @@ struct gmem_inode {
 	const struct guest_memfd_provider_operations *provider_ops;
 	const struct kvm_gmem_backing_ops *backing_ops;
 
+	struct guest_memfd_device device;
+
 	u64 flags;
 	/*
 	 * Every index in this inode, whether memory is populated or
@@ -92,13 +94,19 @@ static __always_inline struct gmem_inode *GMEM_I(struct inode *inode)
 	return container_of(inode, struct gmem_inode, vfs_inode);
 }
 
+static bool gmem_provider_is_device(struct gmem_inode *gi)
+{
+	return gi->provider_ops && gi->provider_ops->device_memory;
+}
+
 static bool kvm_gmem_is_pfn_mode(struct gmem_inode *gi)
 {
 	return gi->flags & GUEST_MEMFD_FLAG_PFN_MODE;
 }
 
 static void *gmem_native_attach(struct inode *gmem_inode,
-				struct file *resource_file)
+				struct file *resource_file,
+				struct guest_memfd_device *device)
 {
 	return GMEM_I(gmem_inode);
 }
@@ -123,7 +131,7 @@ static const struct guest_memfd_provider_operations gmem_native_provider_ops = {
 	.alloc_folio = gmem_native_alloc_folio,
 };
 
-static bool gmem_provider_ops_valid(struct gmem_inode *gi,
+static bool gmem_provider_ops_valid(struct kvm *kvm, struct gmem_inode *gi,
 				     const struct guest_memfd_provider_operations *ops)
 {
 	if (!ops || !ops->attach)
@@ -132,16 +140,27 @@ static bool gmem_provider_ops_valid(struct gmem_inode *gi,
 	if (!!ops->lock != !!ops->unlock)
 		return false;
 
+	if (ops->device_memory) {
+		if (!kvm_gmem_is_pfn_mode(gi) ||
+		    !(gi->flags & GUEST_MEMFD_FLAG_INIT_SHARED) ||
+		    !kvm_arch_gmem_device_supported(kvm) ||
+		    !ops->release || ops->alloc_folio || ops->invalidate_folio ||
+		    !ops->prepare_memory_region)
+			return false;
+	}
+
 	if (kvm_gmem_is_pfn_mode(gi))
-		return ops->get_pfn && ops->lookup_pfn_range;
+		return ops->get_pfn && (ops->device_memory || ops->lookup_pfn_range);
 
 	return ops->alloc_folio != NULL;
 }
 
 #define GMEM_RESOURCE_FD_NATIVE (-1)
 
-static int kvm_gmem_attach_resource(struct inode *inode, int resource_fd)
+static int kvm_gmem_attach_resource(struct kvm *kvm, struct file *file,
+		int resource_fd)
 {
+	struct inode *inode = file_inode(file);
 	struct gmem_inode *gi = GMEM_I(inode);
 	const struct guest_memfd_provider_operations *ops = NULL;
 	void *provider = ERR_PTR(-EOPNOTSUPP);
@@ -158,8 +177,15 @@ static int kvm_gmem_attach_resource(struct inode *inode, int resource_fd)
 			ops = resource_file->f_op->gmem_provider_ops;
 	}
 
-	if (gmem_provider_ops_valid(gi, ops))
-		provider = ops->attach(inode, resource_file);
+	if (gmem_provider_ops_valid(kvm, gi, ops)) {
+		if (ops->device_memory) {
+			gi->device.kvm = kvm;
+			gi->device.size = i_size_read(inode);
+			gi->device.flags = gi->flags;
+		}
+		provider = ops->attach(inode, resource_file,
+				       ops->device_memory ? &gi->device : NULL);
+	}
 
 	if (resource_file)
 		fput(resource_file);
@@ -370,9 +396,15 @@ static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index)
 	return folio;
 }
 
+/*
+ * Select the mapping types that should be covered by generic guest_memfd
+ * invalidation. Device backing can have both shared and private mappings.
+ * For RAM, this depends on the file's initial attributes and whether in-place
+ * conversion allows the same file to back both shared and private mappings.
+ */
 static enum kvm_gfn_range_filter kvm_gmem_get_all_gfns_filter(struct inode *inode)
 {
-	if (gmem_in_place_conversion)
+	if (gmem_provider_is_device(GMEM_I(inode)) || gmem_in_place_conversion)
 		return KVM_FILTER_SHARED | KVM_FILTER_PRIVATE;
 
 	if (GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED)
@@ -675,6 +707,10 @@ static long kvm_gmem_fallocate(struct file *file, int mode, loff_t offset,
 {
 	int ret;
 
+	/* Device MMIO cannot be preallocated or discarded by hole punching. */
+	if (gmem_provider_is_device(GMEM_I(file_inode(file))))
+		return -EOPNOTSUPP;
+
 	if (!(mode & FALLOC_FL_KEEP_SIZE))
 		return -EOPNOTSUPP;
 
@@ -899,7 +935,8 @@ static int kvm_gmem_mmap(struct file *file, struct vm_area_struct *vma)
 {
 	struct gmem_inode *gi = GMEM_I(file_inode(file));
 
-	if (!kvm_gmem_supports_mmap(file_inode(file)))
+	if (gmem_provider_is_device(gi) ||
+	    !kvm_gmem_supports_mmap(file_inode(file)))
 		return -ENODEV;
 
 	if ((vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) !=
@@ -1444,6 +1481,16 @@ static int kvm_gmem_init_inode(struct inode *inode, loff_t size, u64 flags)
 	return r;
 }
 
+bool __weak kvm_arch_gmem_device_supported(struct kvm *kvm)
+{
+	return false;
+}
+
+void __weak kvm_arch_gmem_device_bind(struct kvm_memory_slot *slot,
+		const struct guest_memfd_device_binding *binding)
+{
+}
+
 static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags, int resource_fd)
 {
 	static const char *name = "[kvm-gmem]";
@@ -1499,7 +1546,7 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags, int resour
 	 */
 	smp_wmb();
 
-	err = kvm_gmem_attach_resource(inode, resource_fd);
+	err = kvm_gmem_attach_resource(kvm, file, resource_fd);
 	if (err) {
 		fput(file);
 		put_unused_fd(fd);
@@ -1591,6 +1638,16 @@ int kvm_gmem_prepare_memory_region(struct kvm *kvm, struct kvm_memory_slot *slot
 		}
 	}
 
+	/*
+	 * Device slots resolve both shared and private faults through guest_memfd.
+	 * Identify them as device memory and pass the provider's binding to the
+	 * architecture so it can associate mappings with the protected device.
+	 */
+	if (gmem_provider_is_device(gi)) {
+		slot->flags |= KVM_MEMSLOT_GMEM_ONLY | KVM_MEMSLOT_GMEM_DEVICE;
+		kvm_arch_gmem_device_bind(slot, &gi->device.binding);
+	}
+
 	/*
 	 * memslots of flag KVM_MEM_GUEST_MEMFD are immutable to change, so
 	 * kvm_gmem_bind() must occur on a new memslot.  Because the memslot
@@ -1788,6 +1845,10 @@ static long __kvm_gmem_populate(struct kvm *kvm, struct kvm_memory_slot *slot,
 	kvm_pfn_t pfn;
 	int ret;
 
+	/* Device MMIO cannot be initialized through the RAM population callback. */
+	if (slot->flags & KVM_MEMSLOT_GMEM_DEVICE)
+		return -EINVAL;
+
 	filemap_invalidate_lock(file->f_mapping);
 
 	folio = __kvm_gmem_get_pfn(file, slot, index, &pfn, NULL);
@@ -1915,6 +1976,7 @@ static struct inode *kvm_gmem_alloc_inode(struct super_block *sb)
 	gi->provider = NULL;
 	gi->provider_ops = NULL;
 	gi->backing_ops = NULL;
+	memset(&gi->device, 0, sizeof(gi->device));
 	INIT_LIST_HEAD(&gi->gmem_file_list);
 	return &gi->vfs_inode;
 }
@@ -1926,8 +1988,13 @@ static void kvm_gmem_destroy_inode(struct inode *inode)
 	/*
 	 * Skip if the provider was not yet set up, as iput() can reach this on
 	 * error paths before provider_ops is initialized.
+	 *
+	 * Device PFNs describe MMIO, with no RAM pages to reclaim or return to
+	 * the provider. Protected device mappings require architecture device
+	 * unmap instead of this RAM PFN cleanup.
 	 */
-	if (gi->provider_ops && kvm_gmem_is_pfn_mode(gi)) {
+	if (gi->provider_ops && kvm_gmem_is_pfn_mode(gi) &&
+	    !gmem_provider_is_device(gi)) {
 		pgoff_t nr_pages = i_size_read(inode) >> PAGE_SHIFT;
 
 		kvm_gmem_pfn_reclaim(gi, 0, nr_pages);
-- 
2.43.0

[RFC PATCH v4 07/25] KVM: guest_memfd: Support private/shared conversion of device memory

[Aneesh Kumar K.V (Arm)]

Allow device-backed guest_memfd ranges to be converted between shared
and private memory using file offsets. Use the common attribute conversion
path for inode locking, maple-tree allocation, guest invalidation and
attribute publication.

Reuse PFN attribute preparation to revoke host mappings before private
conversion. Resolve guest addresses through memslot bindings and obtain
physical addresses through the provider's get_pfn() callback.

Install private device mappings during attribute conversion because
device mapping validation needs them before the guest's request is
completed. That validation does not require a preceding guest PFN
lookup, so a prepare_pfn() callback cannot be the sole place that
installs mappings. Use the architecture device-map hook before
publishing private attributes, independently of guest faults. The VMM
validates guest MMIO requests against its BAR layout before setting the
corresponding range private.

Sharing also completes before publishing attributes. Generic
invalidation with KVM_FILTER_PRIVATE has already removed the protected
device mappings, so the device conversion callback needs no additional
shared operation. Shared host and guest mappings can be installed by
their normal fault paths.

Leave revoke() and populate() unset, so full revocation, hole punching
and preallocation remain unsupported at this point in the series.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 Documentation/virt/kvm/api.rst |   5 ++
 include/linux/kvm_host.h       |   2 +
 virt/kvm/guest_memfd.c         | 103 ++++++++++++++++++++++++++++++++-
 3 files changed, 107 insertions(+), 3 deletions(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index 8455a9c7c971..4482d34ec281 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -6540,6 +6540,11 @@ provider. PFN_MODE selects the PFN callbacks; providers that supply device
 operations additionally require INIT_SHARED and a VM supporting protected
 device mappings.
 
+KVM_SET_MEMORY_ATTRIBUTES2 converts device memory using guest_memfd offsets.
+For private conversion, KVM resolves the guest address through the memslot
+binding and obtains the physical address from the provider. Shared conversion
+removes the protected device mappings before publishing shared attributes.
+
 When the KVM MMU performs a PFN lookup to service a guest fault, the fault will
 always be consumed from guest_memfd, regardless of whether it is a shared or a
 private fault (unless in-place conversion is disabled and the backing
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index f2dd387ab0fa..c7831356b913 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -2618,6 +2618,8 @@ struct guest_memfd_device_binding;
 bool kvm_arch_gmem_device_supported(struct kvm *kvm);
 void kvm_arch_gmem_device_bind(struct kvm_memory_slot *slot,
 		const struct guest_memfd_device_binding *binding);
+int kvm_arch_gmem_device_map(struct kvm *kvm, const struct kvm_memory_slot *slot,
+			     u64 gpa, u64 size, u64 pa);
 bool kvm_gmem_is_private_gfn(struct kvm *kvm, gfn_t gfn);
 bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm);
 
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 397ead84bdb3..a06abb6dda82 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -99,6 +99,9 @@ static bool gmem_provider_is_device(struct gmem_inode *gi)
 	return gi->provider_ops && gi->provider_ops->device_memory;
 }
 
+static int kvm_gmem_device_convert_attributes(struct inode *inode, pgoff_t start,
+					      pgoff_t end, u64 attrs);
+
 static bool kvm_gmem_is_pfn_mode(struct gmem_inode *gi)
 {
 	return gi->flags & GUEST_MEMFD_FLAG_PFN_MODE;
@@ -1158,8 +1161,8 @@ static int kvm_gmem_folio_prepare_attributes(struct inode *inode, pgoff_t start,
 	return 0;
 }
 
-static int kvm_gmem_ram_pfn_prepare_attributes(struct inode *inode, pgoff_t start,
-		size_t nr_pages, u64 attrs, pgoff_t *err_index)
+static int kvm_gmem_pfn_prepare_attributes(struct inode *inode, pgoff_t start,
+					   size_t nr_pages, u64 attrs, pgoff_t *err_index)
 {
 	if (!(attrs & KVM_MEMORY_ATTRIBUTE_PRIVATE))
 		return 0;
@@ -1188,14 +1191,22 @@ static const struct kvm_gmem_backing_ops kvm_gmem_folio_backing_ops = {
 static const struct kvm_gmem_backing_ops kvm_gmem_ram_pfn_backing_ops = {
 	.populate = kvm_gmem_ram_pfn_populate,
 	.revoke = kvm_gmem_ram_pfn_revoke,
-	.prepare_attributes = kvm_gmem_ram_pfn_prepare_attributes,
+	.prepare_attributes = kvm_gmem_pfn_prepare_attributes,
 	.convert_attributes = kvm_gmem_ram_pfn_convert_attributes,
 	.prepare_pfn = kvm_gmem_ram_prepare_pfn,
 };
 
+static const struct kvm_gmem_backing_ops kvm_gmem_device_backing_ops = {
+	.prepare_attributes = kvm_gmem_pfn_prepare_attributes,
+	.convert_attributes = kvm_gmem_device_convert_attributes,
+};
+
 static const struct kvm_gmem_backing_ops *
 kvm_gmem_select_backing_ops(struct gmem_inode *gi)
 {
+	if (gmem_provider_is_device(gi))
+		return &kvm_gmem_device_backing_ops;
+
 	if (kvm_gmem_is_pfn_mode(gi))
 		return &kvm_gmem_ram_pfn_backing_ops;
 
@@ -1491,6 +1502,92 @@ void __weak kvm_arch_gmem_device_bind(struct kvm_memory_slot *slot,
 {
 }
 
+int __weak kvm_arch_gmem_device_map(struct kvm *kvm,
+				    const struct kvm_memory_slot *slot,
+				    u64 gpa, u64 size, u64 pa)
+{
+	return -EOPNOTSUPP;
+}
+
+/*
+ * Install protected device mappings for the provider PFNs when making a range
+ * private. Protected device-mapping validation needs these mappings in place
+ * before the guest's request is completed. That validation does not require a
+ * preceding kvm_gmem_get_pfn() call, so prepare_pfn() cannot be the sole place
+ * that installs them. Install them here with kvm_arch_gmem_device_map() before
+ * publishing private attributes, so validation can use the existing mappings.
+ *
+ * Sharing also completes before publishing attributes, but needs no additional
+ * operation here. kvm_gmem_invalidate_start() has already removed the private
+ * device mappings with KVM_FILTER_PRIVATE. Shared host and guest mappings can
+ * then be installed by their normal fault paths.
+ *
+ * Caller holds inode->i_mapping->invalidate_lock for write, which protects
+ * the attributes and offset-to-GPA bindings. The gmem_provider_lock()
+ * prevents device unlock from racing with conversion and publication of
+ * attributes.
+ */
+static int kvm_gmem_device_convert_attributes(struct inode *inode, pgoff_t start,
+					      pgoff_t end, u64 attrs)
+{
+	struct gmem_inode *gi = GMEM_I(inode);
+	struct gmem_file *f;
+	void *entry;
+
+	MA_STATE(mas, &gi->attributes, start, start);
+
+	if (!(attrs & KVM_MEMORY_ATTRIBUTE_PRIVATE))
+		return 0;
+
+	mas_for_each(&mas, entry, end - 1) {
+		pgoff_t first = max(start, mas.index);
+		pgoff_t last = min(end, mas.last + 1);
+
+		if (kvm_gmem_get_attributes(inode, entry) & KVM_MEMORY_ATTRIBUTE_PRIVATE)
+			continue;
+
+		while (first < last) {
+			struct kvm_memory_slot *slot = NULL;
+			kvm_pfn_t pfn, next_pfn;
+			pgoff_t high, next;
+			u64 gpa;
+			int ret;
+
+			kvm_gmem_for_each_file(f, inode) {
+				slot = xa_load(&f->bindings, first);
+				if (slot)
+					break;
+			}
+			if (!slot)
+				return -EINVAL;
+
+			high = min(last, slot->gmem.pgoff + slot->npages);
+			gpa = (slot->base_gfn + first - slot->gmem.pgoff) << PAGE_SHIFT;
+			ret = gmem_provider_get_pfn(gi, first, &pfn, NULL);
+			if (ret)
+				return ret;
+
+			/* Map a contiguous provider PFN run within this binding. */
+			for (next = first + 1; next < high; next++) {
+				ret = gmem_provider_get_pfn(gi, next, &next_pfn, NULL);
+				if (ret)
+					return ret;
+				if (next_pfn != pfn + next - first)
+					break;
+			}
+			high = next;
+			ret = kvm_arch_gmem_device_map(gi->device.kvm, slot, gpa,
+						       (u64)(high - first) << PAGE_SHIFT,
+						       PFN_PHYS(pfn));
+			if (ret)
+				return ret;
+
+			first = high;
+		}
+	}
+	return 0;
+}
+
 static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags, int resource_fd)
 {
 	static const char *name = "[kvm-gmem]";
-- 
2.43.0

[RFC PATCH v4 08/25] KVM: guest_memfd: Track private device ranges after attribute updates

[Aneesh Kumar K.V (Arm)]

The following device access callbacks need to report whether any private
ranges remain without acquiring the inode invalidate lock under a
provider lock.

Add has_private_ranges to struct guest_memfd_device as a cached summary
of the committed attributes. Add an optional update_attributes() backing
operation and call it after storing new attributes on successful
conversion. Device backing scans the attribute tree to refresh the
summary; folio and RAM PFN backing leave the operation unset.

Providers read the summary directly with READ_ONCE(). Device unlock
checks hold the same lock as the provider lock()/unlock() callbacks
to serialize with conversion.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 include/linux/guest_memfd.h |  5 +++++
 virt/kvm/guest_memfd.c      | 27 ++++++++++++++++++++++++++-
 2 files changed, 31 insertions(+), 1 deletion(-)

diff --git a/include/linux/guest_memfd.h b/include/linux/guest_memfd.h
index 6bf44c62b7fe..9869e2ffb4a2 100644
--- a/include/linux/guest_memfd.h
+++ b/include/linux/guest_memfd.h
@@ -24,12 +24,17 @@ struct guest_memfd_device_binding {
  * @flags: GUEST_MEMFD_FLAG_* creation flags
  * @binding: Mapping owner and device ID set by attach(), with the underlying
  *           resources pinned until release()
+ * @has_private_ranges: Core-maintained summary of committed private ranges;
+ *                      providers read it with READ_ONCE(). Device unlock
+ *                      checks must hold the same lock as the provider's
+ *                      lock()/unlock() callbacks to serialize with conversion
  */
 struct guest_memfd_device {
 	struct kvm *kvm;
 	u64 size;
 	u64 flags;
 	struct guest_memfd_device_binding binding;
+	bool has_private_ranges;
 };
 
 /**
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index a06abb6dda82..54b387d296a0 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -63,6 +63,8 @@ struct kvm_gmem_backing_ops {
 	 */
 	int (*prepare_pfn)(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
 			   kvm_pfn_t nr_pages, u64 attrs);
+	/* Called after the common code stores the new attributes. */
+	void (*update_attributes)(struct inode *inode);
 };
 
 struct gmem_inode {
@@ -99,6 +101,7 @@ static bool gmem_provider_is_device(struct gmem_inode *gi)
 	return gi->provider_ops && gi->provider_ops->device_memory;
 }
 
+static void kvm_gmem_device_update_attributes(struct inode *inode);
 static int kvm_gmem_device_convert_attributes(struct inode *inode, pgoff_t start,
 					      pgoff_t end, u64 attrs);
 
@@ -1199,6 +1202,7 @@ static const struct kvm_gmem_backing_ops kvm_gmem_ram_pfn_backing_ops = {
 static const struct kvm_gmem_backing_ops kvm_gmem_device_backing_ops = {
 	.prepare_attributes = kvm_gmem_pfn_prepare_attributes,
 	.convert_attributes = kvm_gmem_device_convert_attributes,
+	.update_attributes = kvm_gmem_device_update_attributes,
 };
 
 static const struct kvm_gmem_backing_ops *
@@ -1265,8 +1269,11 @@ static int __kvm_gmem_set_attributes(struct inode *inode, pgoff_t start,
 	kvm_gmem_invalidate_start(inode, start, end, filter);
 
 	r = kvm_gmem_convert_attributes(inode, start, end, attrs);
-	if (!r)
+	if (!r) {
 		mas_store_prealloc(&mas, xa_mk_value(attrs));
+		if (gi->backing_ops->update_attributes)
+			gi->backing_ops->update_attributes(inode);
+	}
 
 	if (r)
 		mas_destroy(&mas);
@@ -1509,6 +1516,24 @@ int __weak kvm_arch_gmem_device_map(struct kvm *kvm,
 	return -EOPNOTSUPP;
 }
 
+static void kvm_gmem_device_update_attributes(struct inode *inode)
+{
+	struct gmem_inode *gi = GMEM_I(inode);
+	void *entry;
+	bool private = false;
+
+	MA_STATE(mas, &gi->attributes, 0, 0);
+
+	mas_for_each(&mas, entry, ULONG_MAX) {
+		if (kvm_gmem_get_attributes(&gi->vfs_inode, entry) &
+		    KVM_MEMORY_ATTRIBUTE_PRIVATE) {
+			private = true;
+			break;
+		}
+	}
+	WRITE_ONCE(gi->device.has_private_ranges, private);
+}
+
 /*
  * Install protected device mappings for the provider PFNs when making a range
  * private. Protected device-mapping validation needs these mappings in place
-- 
2.43.0

[RFC PATCH v4 09/25] KVM: guest_memfd: Support range revocation of device mappings

[Aneesh Kumar K.V (Arm)]

Give device backing a revoke operation that removes host and both shared
and private guest translations under provider exclusion, without reclaiming
MMIO PFNs or changing attributes. Keep fallocate and hole punching
unsupported for device memory.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 virt/kvm/guest_memfd.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 54b387d296a0..4c9ac047d06f 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -543,6 +543,18 @@ static int kvm_gmem_ram_pfn_revoke(struct inode *inode, pgoff_t start, pgoff_t e
 	return 0;
 }
 
+static int kvm_gmem_device_revoke(struct inode *inode, pgoff_t start, pgoff_t end,
+		const unsigned long *pfn, bool put_pfn_range)
+{
+	/*
+	 * Common invalidation handles shared and private guest mappings. Only
+	 * host userspace mappings need to be unmapped here; MMIO has no RAM
+	 * pages to reclaim or provider allocations to release.
+	 */
+	unmap_mapping_pages(inode->i_mapping, start, end - start, true);
+	return 0;
+}
+
 static int __kvm_gmem_revoke(struct inode *inode, pgoff_t start,
 			      size_t nr_pages, const unsigned long *pfn,
 			      bool put_pfn_range)
@@ -1200,6 +1212,7 @@ static const struct kvm_gmem_backing_ops kvm_gmem_ram_pfn_backing_ops = {
 };
 
 static const struct kvm_gmem_backing_ops kvm_gmem_device_backing_ops = {
+	.revoke = kvm_gmem_device_revoke,
 	.prepare_attributes = kvm_gmem_pfn_prepare_attributes,
 	.convert_attributes = kvm_gmem_device_convert_attributes,
 	.update_attributes = kvm_gmem_device_update_attributes,
-- 
2.43.0

[RFC PATCH v4 10/25] KVM: guest_memfd: Add shared device mmap through PFN providers

[Aneesh Kumar K.V (Arm)]

Map shared device memory through the existing get_pfn callback. Set
device VMA flags and protections in the core, and retain inode exclusion
through PFN lookup and PTE insertion. Validate BAR access at fault time;
memslot creation validates the complete interval separately.

Private pages remain inaccessible to userspace. Conversion revokes host
mappings before changing attributes. VFIO fd mmap remains excluded.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 include/linux/guest_memfd.h |  6 +++++-
 virt/kvm/guest_memfd.c      | 18 +++++++++++++++---
 2 files changed, 20 insertions(+), 4 deletions(-)

diff --git a/include/linux/guest_memfd.h b/include/linux/guest_memfd.h
index 9869e2ffb4a2..88c9102b21fa 100644
--- a/include/linux/guest_memfd.h
+++ b/include/linux/guest_memfd.h
@@ -2,6 +2,7 @@
 #ifndef _LINUX_GUEST_MEMFD_H
 #define _LINUX_GUEST_MEMFD_H
 
+#include <linux/mm_types.h>
 #include <linux/types.h>
 
 struct file;
@@ -45,7 +46,10 @@ struct guest_memfd_device {
  *          in the device argument, which is NULL for other providers.
  * @release: Release provider context and unpin any resources. Must not access
  *           the VM here; its file reference may be gone.
- * @device_memory: Backing is device MMIO rather than RAM.
+ * @device_memory: Backing is device MMIO rather than RAM. The core maps its
+ *                 userspace VMAs noncached and decrypted, with device flags.
+ *                 Shared userspace faults are resolved through @get_pfn, with
+ *                 the same error handling as RAM providers.
  * @prepare_memory_region: Optional check of a memory range and preparation of
  *                         access to its backing memory. Called after common
  *                         file checks and before publishing the memslot.
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 4c9ac047d06f..dd567979d938 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -949,12 +949,16 @@ static const struct vm_operations_struct kvm_gmem_vm_ops = {
 #endif
 };
 
+static const struct vm_operations_struct kvm_gmem_device_vm_ops = {
+	.fault = kvm_gmem_fault_user_mapping,
+};
+
 static int kvm_gmem_mmap(struct file *file, struct vm_area_struct *vma)
 {
-	struct gmem_inode *gi = GMEM_I(file_inode(file));
+	struct inode *inode = file_inode(file);
+	struct gmem_inode *gi = GMEM_I(inode);
 
-	if (gmem_provider_is_device(gi) ||
-	    !kvm_gmem_supports_mmap(file_inode(file)))
+	if (!kvm_gmem_supports_mmap(inode))
 		return -ENODEV;
 
 	if ((vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) !=
@@ -965,6 +969,14 @@ static int kvm_gmem_mmap(struct file *file, struct vm_area_struct *vma)
 	if (kvm_gmem_is_pfn_mode(gi))
 		vm_flags_set(vma, VM_PFNMAP);
 
+	if (gmem_provider_is_device(gi)) {
+		vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
+		vma->vm_page_prot = pgprot_decrypted(vma->vm_page_prot);
+		vm_flags_set(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP);
+		vma->vm_ops = &kvm_gmem_device_vm_ops;
+		return 0;
+	}
+
 	vma->vm_ops = &kvm_gmem_vm_ops;
 
 	return 0;
-- 
2.43.0

[RFC PATCH v4 11/25] KVM: guest_memfd: Clear private device state on final close

[Aneesh Kumar K.V (Arm)]

Add an optional backing release() callback after the generic
file-release path invalidates guest mappings. Device backing uses it to
clear has_private_ranges so device cleanup can proceed after the last
file user is gone.

Hold the provider lock across invalidation and the release callback so
device unlock cannot race with mapping removal and summary updates.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 virt/kvm/guest_memfd.c | 26 +++++++++++++++++++++++---
 1 file changed, 23 insertions(+), 3 deletions(-)

diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index dd567979d938..d3de2e7bb9f5 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -65,6 +65,12 @@ struct kvm_gmem_backing_ops {
 			   kvm_pfn_t nr_pages, u64 attrs);
 	/* Called after the common code stores the new attributes. */
 	void (*update_attributes)(struct inode *inode);
+	/*
+	 * Finish file cleanup after invalidation. Caller holds kvm->slots_lock
+	 * and inode->i_mapping->invalidate_lock for write, between calls to
+	 * gmem_provider_lock() and gmem_provider_unlock().
+	 */
+	void (*release)(struct inode *inode);
 };
 
 struct gmem_inode {
@@ -104,6 +110,7 @@ static bool gmem_provider_is_device(struct gmem_inode *gi)
 static void kvm_gmem_device_update_attributes(struct inode *inode);
 static int kvm_gmem_device_convert_attributes(struct inode *inode, pgoff_t start,
 					      pgoff_t end, u64 attrs);
+static void kvm_gmem_device_release(struct inode *inode);
 
 static bool kvm_gmem_is_pfn_mode(struct gmem_inode *gi)
 {
@@ -751,9 +758,11 @@ static long kvm_gmem_fallocate(struct file *file, int mode, loff_t offset,
 static int kvm_gmem_release(struct inode *inode, struct file *file)
 {
 	struct gmem_file *f = file->private_data;
+	struct gmem_inode *gi = GMEM_I(inode);
 	struct kvm_memory_slot *slot;
 	struct kvm *kvm = f->kvm;
 	unsigned long index;
+	enum kvm_gfn_range_filter filter = kvm_gmem_get_all_gfns_filter(inode);
 
 	/*
 	 * Prevent concurrent attempts to *unbind* a memslot.  This is the last
@@ -766,6 +775,8 @@ static int kvm_gmem_release(struct inode *inode, struct file *file)
 
 	filemap_invalidate_lock(inode->i_mapping);
 
+	gmem_provider_lock(gi);
+
 	/*
 	 * Note!  synchronize_srcu() is _not_ needed after nullifying memslot
 	 * bindings as slot->gmem.file cannot be set back to a non-null value
@@ -815,13 +826,15 @@ static int kvm_gmem_release(struct inode *inode, struct file *file)
 		WRITE_ONCE(slot->gmem.file, NULL);
 
 	/*
-	 * All in-flight operations are gone and new bindings can be created.
+	 * All in-flight operations are gone and no new bindings can be created.
 	 * Zap all SPTEs pointed at by this file.  Do not free the backing
 	 * memory, as its lifetime is associated with the inode, not the file.
 	 */
-	__kvm_gmem_invalidate_start(f, 0, -1ul,
-				    kvm_gmem_get_all_gfns_filter(inode));
+	__kvm_gmem_invalidate_start(f, 0, -1ul, filter);
 	__kvm_gmem_invalidate_end(f, 0, -1ul);
+	if (gi->backing_ops && gi->backing_ops->release)
+		gi->backing_ops->release(inode);
+	gmem_provider_unlock(gi);
 
 	list_del(&f->entry);
 
@@ -1228,6 +1241,7 @@ static const struct kvm_gmem_backing_ops kvm_gmem_device_backing_ops = {
 	.prepare_attributes = kvm_gmem_pfn_prepare_attributes,
 	.convert_attributes = kvm_gmem_device_convert_attributes,
 	.update_attributes = kvm_gmem_device_update_attributes,
+	.release = kvm_gmem_device_release,
 };
 
 static const struct kvm_gmem_backing_ops *
@@ -1638,6 +1652,12 @@ static int kvm_gmem_device_convert_attributes(struct inode *inode, pgoff_t start
 	return 0;
 }
 
+/* No file users remain, so private attributes cannot create new mappings. */
+static void kvm_gmem_device_release(struct inode *inode)
+{
+	WRITE_ONCE(GMEM_I(inode)->device.has_private_ranges, false);
+}
+
 static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags, int resource_fd)
 {
 	static const char *name = "[kvm-gmem]";
-- 
2.43.0

[RFC PATCH v4 12/25] KVM: guest_memfd: Invalidate mappings before memslot unbind

[Aneesh Kumar K.V (Arm)]

Use the generic invalidate_start/end helpers to remove shared and
private device mappings before removing the memslot's guest_memfd
binding.

Hold gmem_provider_lock() across invalidation and binding removal to
prevent device unlock from racing with teardown.

Keep the file attributes and has_private_ranges unchanged. Deleting a
memslot removes mappings but does not share the backing range. Device
unlock remains blocked until userspace shares the private ranges or
closes the last guest_memfd reference.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 virt/kvm/guest_memfd.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index d3de2e7bb9f5..870fe11c1c5c 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -1873,6 +1873,8 @@ void kvm_gmem_unbind(struct kvm_memory_slot *slot)
 	unsigned long start = slot->gmem.pgoff;
 	unsigned long end = start + slot->npages;
 	struct gmem_file *f;
+	struct gmem_inode *gi;
+	enum kvm_gfn_range_filter filter;
 
 	/*
 	 * Nothing to do if the underlying file was _already_ closed, as
@@ -1892,8 +1894,13 @@ void kvm_gmem_unbind(struct kvm_memory_slot *slot)
 	 * drops slots_lock, so there's no need to verify the file is live.
 	 */
 	f = file->private_data;
+	gi = GMEM_I(file_inode(file));
+	filter = kvm_gmem_get_all_gfns_filter(file_inode(file));
 
 	filemap_invalidate_lock(file->f_mapping);
+	gmem_provider_lock(gi);
+	__kvm_gmem_invalidate_start(f, start, end, filter);
+	__kvm_gmem_invalidate_end(f, start, end);
 	xa_store_range(&f->bindings, start, end - 1, NULL, GFP_KERNEL);
 
 	/*
@@ -1902,6 +1909,7 @@ void kvm_gmem_unbind(struct kvm_memory_slot *slot)
 	 * to guarantee kvm_gmem_get_pfn() can't see the slot+file.
 	 */
 	WRITE_ONCE(slot->gmem.file, NULL);
+	gmem_provider_unlock(gi);
 	filemap_invalidate_unlock(file->f_mapping);
 }
 
-- 
2.43.0

[RFC PATCH v4 13/25] vfio: Track device use by guest_memfd

[Aneesh Kumar K.V (Arm)]

Add gmem_attached to record whether a device supplies guest_memfd
backing, and initialize gmem_lock.

Changing gmem_attached requires both gmem_lock held for write and
dev_set->lock held. This lets device file operations check the flag
under gmem_lock for read, while reset checks use dev_set->lock.

This patch adds the flag and initializes the lock. Later patches set
the flag after successful provider setup and clear it after provider
cleanup.

Cc: Alex Williamson <alex@shazbot.org>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 drivers/vfio/vfio_main.c | 1 +
 include/linux/vfio.h     | 7 +++++++
 2 files changed, 8 insertions(+)

diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
index 4e5d738ccf4b..ff49f2fdb602 100644
--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -290,6 +290,7 @@ static int vfio_init_device(struct vfio_device *device, struct device *dev,
 		return ret;
 	}
 
+	init_rwsem(&device->gmem_lock);
 	device->index = ret;
 	init_completion(&device->comp);
 	device->dev = dev;
diff --git a/include/linux/vfio.h b/include/linux/vfio.h
index 0cc91c6f96d2..2436b8bd600a 100644
--- a/include/linux/vfio.h
+++ b/include/linux/vfio.h
@@ -37,6 +37,13 @@ struct vfio_device_set {
 };
 
 struct vfio_device {
+	/*
+	 * Serializes guest_memfd provider setup and teardown with device file
+	 * operations.
+	 */
+	struct rw_semaphore gmem_lock;
+	/* Updates hold gmem_lock for write and dev_set->lock; either permits reads. */
+	bool gmem_attached;
 	struct device *dev;
 	const struct vfio_device_ops *ops;
 	/*
-- 
2.43.0

[RFC PATCH v4 14/25] vfio/pci: Add helpers to check and serialize BAR state changes

[Aneesh Kumar K.V (Arm)]

Add vfio_pci_begin_bar_state_change() to check whether a BAR state
change is allowed before taking vdev->memory_lock for write and removing
existing BAR mappings with vfio_pci_zap_bars(). Return -EBUSY if the
check fails, before removing mappings or changing hardware.

Keep memory_lock held on success until the caller finishes the hardware
operation and calls vfio_pci_end_bar_state_change().

vfio_pci_bar_state_change_allowed() initially returns true. Later
patches use these helpers for command-register writes, reset and power
changes, then reject those operations while the device supplies
guest_memfd backing.

Cc: Alex Williamson <alex@shazbot.org>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 drivers/vfio/pci/vfio_pci_core.c | 15 +++++++++++++++
 drivers/vfio/pci/vfio_pci_priv.h | 10 ++++++++++
 2 files changed, 25 insertions(+)

diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index fcbb2da860e2..99778cf0cc3c 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -1709,6 +1709,21 @@ void vfio_pci_zap_and_down_write_memory_lock(struct vfio_pci_core_device *vdev)
 	vfio_pci_zap_bars(vdev);
 }
 
+int vfio_pci_begin_bar_state_change(struct vfio_pci_core_device *vdev)
+{
+	if (!vfio_pci_bar_state_change_allowed(vdev))
+		return -EBUSY;
+
+	down_write(&vdev->memory_lock);
+	vfio_pci_zap_bars(vdev);
+	return 0;
+}
+
+void vfio_pci_end_bar_state_change(struct vfio_pci_core_device *vdev)
+{
+	up_write(&vdev->memory_lock);
+}
+
 u16 vfio_pci_memory_lock_and_enable(struct vfio_pci_core_device *vdev)
 {
 	u16 cmd;
diff --git a/drivers/vfio/pci/vfio_pci_priv.h b/drivers/vfio/pci/vfio_pci_priv.h
index 4e7162234a2e..0af93f6d9a8a 100644
--- a/drivers/vfio/pci/vfio_pci_priv.h
+++ b/drivers/vfio/pci/vfio_pci_priv.h
@@ -11,6 +11,13 @@
 /* Cap maximum number of ioeventfds per device (arbitrary) */
 #define VFIO_PCI_IOEVENTFD_MAX		1000
 
+/* Later patches reject BAR state changes while guest_memfd uses the device. */
+static inline bool
+vfio_pci_bar_state_change_allowed(struct vfio_pci_core_device *vdev)
+{
+	return true;
+}
+
 struct vfio_pci_ioeventfd {
 	struct list_head	next;
 	struct vfio_pci_core_device	*vdev;
@@ -69,6 +76,9 @@ int vfio_pci_set_power_state(struct vfio_pci_core_device *vdev,
 			     pci_power_t state);
 
 void vfio_pci_zap_and_down_write_memory_lock(struct vfio_pci_core_device *vdev);
+/* Success retains memory_lock through the hardware operation. */
+int vfio_pci_begin_bar_state_change(struct vfio_pci_core_device *vdev);
+void vfio_pci_end_bar_state_change(struct vfio_pci_core_device *vdev);
 u16 vfio_pci_memory_lock_and_enable(struct vfio_pci_core_device *vdev);
 void vfio_pci_memory_unlock_and_restore(struct vfio_pci_core_device *vdev,
 					u16 cmd);
-- 
2.43.0

[RFC PATCH v4 15/25] vfio/pci: Check command enable changes before updating device state

[Aneesh Kumar K.V (Arm)]

Add checks in vfio_basic_config_write() for changes to Bus Master Enable
and Memory Enable. Move the existing acquisition of vdev->memory_lock
before reading PCI_COMMAND so the read, checks and updates all occur
while the lock is held for write.

Read Bus Master Enable from hardware and Memory Enable from vconfig,
which also tracks the setting for VFs. If vfio_pci_bar_state_change_allowed()
returns false, reject changes to either bit with -EBUSY before updating
hardware or vconfig. Allow writes that leave both bits unchanged.

Memory Enable controls BAR decoding, while Bus Master Enable controls
device-initiated DMA. Later patches reject changes to both while the
device supplies guest_memfd backing. The check currently returns true,
preserving existing behavior.

Cc: Alex Williamson <alex@shazbot.org>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 drivers/vfio/pci/vfio_pci_config.c | 64 +++++++++++++++++++++---------
 1 file changed, 46 insertions(+), 18 deletions(-)

diff --git a/drivers/vfio/pci/vfio_pci_config.c b/drivers/vfio/pci/vfio_pci_config.c
index 9914f3ac69ae..5b97f64ab3ba 100644
--- a/drivers/vfio/pci/vfio_pci_config.c
+++ b/drivers/vfio/pci/vfio_pci_config.c
@@ -568,19 +568,49 @@ static int vfio_basic_config_write(struct vfio_pci_core_device *vdev, int pos,
 	struct pci_dev *pdev = vdev->pdev;
 	__le16 *virt_cmd;
 	u16 new_cmd = 0;
+	bool zap_bars_required = false;
 	int ret;
 
 	virt_cmd = (__le16 *)&vdev->vconfig[PCI_COMMAND];
 
 	if (offset == PCI_COMMAND) {
 		bool phys_mem, virt_mem, new_mem, phys_io, virt_io, new_io;
-		u16 phys_cmd;
+		bool enable_bits_changed;
+		u16 phys_cmd, old_cmd;
+
+		new_cmd = le32_to_cpu(val);
+		new_mem = !!(new_cmd & PCI_COMMAND_MEMORY);
+		zap_bars_required =
+			vfio_pci_bar_state_change_allowed(vdev) && !new_mem;
+
+		if (zap_bars_required) {
+			ret = vfio_pci_begin_bar_state_change(vdev);
+			if (ret)
+				return ret;
+			vfio_pci_dma_buf_move(vdev, true);
+		} else {
+			down_write(&vdev->memory_lock);
+		}
 
 		ret = pci_user_read_config_word(pdev, PCI_COMMAND, &phys_cmd);
-		if (ret)
-			return ret;
+		if (ret) {
+			count = ret;
+			goto out_unlock;
+		}
 
-		new_cmd = le32_to_cpu(val);
+		/*
+		 * Check BME and Memory Enable changes while holding memory_lock
+		 * for write, before updating hardware or vconfig. Return an
+		 * error rather than silently masking a rejected write.
+		 */
+		old_cmd = (phys_cmd & ~PCI_COMMAND_MEMORY) |
+			  (le16_to_cpu(*virt_cmd) & PCI_COMMAND_MEMORY);
+		enable_bits_changed = (old_cmd ^ new_cmd) &
+				      (PCI_COMMAND_MASTER | PCI_COMMAND_MEMORY);
+		if (enable_bits_changed && !vfio_pci_bar_state_change_allowed(vdev)) {
+			count = -EBUSY;
+			goto out_unlock;
+		}
 
 		phys_io = !!(phys_cmd & PCI_COMMAND_IO);
 		virt_io = !!(le16_to_cpu(*virt_cmd) & PCI_COMMAND_IO);
@@ -588,14 +618,6 @@ static int vfio_basic_config_write(struct vfio_pci_core_device *vdev, int pos,
 
 		phys_mem = !!(phys_cmd & PCI_COMMAND_MEMORY);
 		virt_mem = !!(le16_to_cpu(*virt_cmd) & PCI_COMMAND_MEMORY);
-		new_mem = !!(new_cmd & PCI_COMMAND_MEMORY);
-
-		if (!new_mem) {
-			vfio_pci_zap_and_down_write_memory_lock(vdev);
-			vfio_pci_dma_buf_move(vdev, true);
-		} else {
-			down_write(&vdev->memory_lock);
-		}
 
 		/*
 		 * If the user is writing mem/io enable (new_mem/io) and we
@@ -614,11 +636,8 @@ static int vfio_basic_config_write(struct vfio_pci_core_device *vdev, int pos,
 	}
 
 	count = vfio_default_config_write(vdev, pos, count, perm, offset, val);
-	if (count < 0) {
-		if (offset == PCI_COMMAND)
-			up_write(&vdev->memory_lock);
-		return count;
-	}
+	if (count < 0)
+		goto out_unlock;
 
 	/*
 	 * Save current memory/io enable bits in vconfig to allow for
@@ -632,9 +651,18 @@ static int vfio_basic_config_write(struct vfio_pci_core_device *vdev, int pos,
 
 		if (__vfio_pci_memory_enabled(vdev))
 			vfio_pci_dma_buf_move(vdev, false);
-		up_write(&vdev->memory_lock);
 	}
 
+out_unlock:
+	if (offset == PCI_COMMAND) {
+		if (zap_bars_required)
+			vfio_pci_end_bar_state_change(vdev);
+		else
+			up_write(&vdev->memory_lock);
+	}
+	if (count < 0)
+		return count;
+
 	/* Emulate INTx disable */
 	if (offset >= PCI_COMMAND && offset <= PCI_COMMAND + 1) {
 		bool virt_intx_disable;
-- 
2.43.0

[RFC PATCH v4 16/25] vfio/pci: Check whether BAR state changes are allowed before reset

[Aneesh Kumar K.V (Arm)]

Add checks in vfio_pci_ioctl_reset(), vfio_exp_config_write() and
vfio_af_config_write() before resetting the device. Check PCIe and AF
FLR request bits before vfio_default_config_write() updates vconfig, so
a rejected request returns -EBUSY without changing virtual state.

Use vfio_pci_begin_bar_state_change() and
vfio_pci_end_bar_state_change() around function resets. These retain the
existing memory_lock protection while allowing reset to be rejected
before BAR mappings are removed.

In vfio_pci_dev_set_hot_reset(), check every device in
dev_set->device_list while holding the existing dev_set->lock. Reject
the reset before acquiring runtime-PM references for the device set or
removing BAR mappings if any device fails the check.

vfio_pci_bar_state_change_allowed() currently returns true. Later
patches reject resets while a device supplies guest_memfd backing.

Cc: Alex Williamson <alex@shazbot.org>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 drivers/vfio/pci/vfio_pci_config.c | 38 ++++++++++++++++++++++++++----
 drivers/vfio/pci/vfio_pci_core.c   | 20 ++++++++++++++--
 2 files changed, 52 insertions(+), 6 deletions(-)

diff --git a/drivers/vfio/pci/vfio_pci_config.c b/drivers/vfio/pci/vfio_pci_config.c
index 5b97f64ab3ba..537e7f8bc884 100644
--- a/drivers/vfio/pci/vfio_pci_config.c
+++ b/drivers/vfio/pci/vfio_pci_config.c
@@ -733,6 +733,24 @@ static int __init init_pci_cap_basic_perm(struct perm_bits *perm)
 	return 0;
 }
 
+/* Config accesses are split into aligned byte, word or dword chunks. */
+static bool vfio_pcie_flr_requested(int offset, int count, __le32 val)
+{
+	if (offset == PCI_EXP_DEVCTL && count >= 2)
+		return le32_to_cpu(val) & PCI_EXP_DEVCTL_BCR_FLR;
+
+	if (offset == PCI_EXP_DEVCTL + 1)
+		return le32_to_cpu(val) & (PCI_EXP_DEVCTL_BCR_FLR >> 8);
+
+	return false;
+}
+
+static bool vfio_af_flr_requested(int offset, __le32 val)
+{
+	return offset == PCI_AF_CTRL &&
+	       (le32_to_cpu(val) & PCI_AF_CTRL_FLR);
+}
+
 /*
  * It takes all the required locks to protect the access of power related
  * variables and then invokes vfio_pci_set_power_state().
@@ -915,6 +933,10 @@ static int vfio_exp_config_write(struct vfio_pci_core_device *vdev, int pos,
 				  offset + PCI_EXP_DEVCTL);
 	int readrq = le16_to_cpu(*ctrl) & PCI_EXP_DEVCTL_READRQ;
 
+	if (vfio_pcie_flr_requested(offset, count, val) &&
+	    !vfio_pci_bar_state_change_allowed(vdev))
+		return -EBUSY;
+
 	count = vfio_default_config_write(vdev, pos, count, perm, offset, val);
 	if (count < 0)
 		return count;
@@ -936,12 +958,14 @@ static int vfio_exp_config_write(struct vfio_pci_core_device *vdev, int pos,
 						 &cap);
 
 		if (!ret && (cap & PCI_EXP_DEVCAP_FLR)) {
-			vfio_pci_zap_and_down_write_memory_lock(vdev);
+			ret = vfio_pci_begin_bar_state_change(vdev);
+			if (ret)
+				return ret;
 			vfio_pci_dma_buf_move(vdev, true);
 			pci_try_reset_function(vdev->pdev);
 			if (__vfio_pci_memory_enabled(vdev))
 				vfio_pci_dma_buf_move(vdev, false);
-			up_write(&vdev->memory_lock);
+			vfio_pci_end_bar_state_change(vdev);
 		}
 	}
 
@@ -1000,6 +1024,10 @@ static int vfio_af_config_write(struct vfio_pci_core_device *vdev, int pos,
 {
 	u8 *ctrl = vdev->vconfig + pos - offset + PCI_AF_CTRL;
 
+	if (vfio_af_flr_requested(offset, val) &&
+	    !vfio_pci_bar_state_change_allowed(vdev))
+		return -EBUSY;
+
 	count = vfio_default_config_write(vdev, pos, count, perm, offset, val);
 	if (count < 0)
 		return count;
@@ -1021,12 +1049,14 @@ static int vfio_af_config_write(struct vfio_pci_core_device *vdev, int pos,
 						&cap);
 
 		if (!ret && (cap & PCI_AF_CAP_FLR) && (cap & PCI_AF_CAP_TP)) {
-			vfio_pci_zap_and_down_write_memory_lock(vdev);
+			ret = vfio_pci_begin_bar_state_change(vdev);
+			if (ret)
+				return ret;
 			vfio_pci_dma_buf_move(vdev, true);
 			pci_try_reset_function(vdev->pdev);
 			if (__vfio_pci_memory_enabled(vdev))
 				vfio_pci_dma_buf_move(vdev, false);
-			up_write(&vdev->memory_lock);
+			vfio_pci_end_bar_state_change(vdev);
 		}
 	}
 
diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index 99778cf0cc3c..f31fbd980f3e 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -1307,7 +1307,9 @@ static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,
 	if (!vdev->reset_works)
 		return -EINVAL;
 
-	vfio_pci_zap_and_down_write_memory_lock(vdev);
+	ret = vfio_pci_begin_bar_state_change(vdev);
+	if (ret)
+		return ret;
 
 	/*
 	 * This function can be invoked while the power state is non-D0. If
@@ -1324,7 +1326,7 @@ static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,
 	ret = pci_try_reset_function(vdev->pdev);
 	if (__vfio_pci_memory_enabled(vdev))
 		vfio_pci_dma_buf_move(vdev, false);
-	up_write(&vdev->memory_lock);
+	vfio_pci_end_bar_state_change(vdev);
 
 	return ret;
 }
@@ -2503,6 +2505,20 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,
 
 	mutex_lock(&dev_set->lock);
 
+	/*
+	 * Check every device before acquiring runtime-PM references for the
+	 * device set or removing BAR mappings. Generic ioctl dispatch already
+	 * holds a PM reference for the target. Holding dev_set->lock prevents
+	 * gmem_attached from changing during these checks and the reset.
+	 */
+	lockdep_assert_held(&dev_set->lock);
+	list_for_each_entry(vdev, &dev_set->device_list, vdev.dev_set_list) {
+		if (!vfio_pci_bar_state_change_allowed(vdev)) {
+			ret = -EBUSY;
+			goto err_unlock;
+		}
+	}
+
 	pdev = vfio_pci_dev_set_resettable(dev_set);
 	if (!pdev) {
 		ret = -EINVAL;
-- 
2.43.0

[RFC PATCH v4 17/25] vfio/pci: Check low-power requests before updating device state

[Aneesh Kumar K.V (Arm)]

In vfio_pm_config_write(), check requests for D1, D2 or D3hot before
vfio_default_config_write() updates vconfig. Return -EBUSY if
vfio_pci_bar_state_change_allowed() returns false. D0 requests and
writes to unrelated PMCSR fields retain their existing handling.

Use vfio_pci_begin_bar_state_change() for D3hot transitions in
vfio_lock_and_set_power_state(), and return its error to the config
write caller. On success, retain the existing behavior

In vfio_pci_runtime_pm_entry(), reject explicit low-power entry before
removing BAR mappings or dropping the runtime-PM reference.

These checks currently allow all requests. Later patches reject
low-power requests while the device supplies guest_memfd backing,
because entering a lower power state can make mapped BAR PFNs
inaccessible.

Cc: Alex Williamson <alex@shazbot.org>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 drivers/vfio/pci/vfio_pci_config.c | 30 ++++++++++++++++++++++++++----
 drivers/vfio/pci/vfio_pci_core.c   | 30 +++++++++++++++++-------------
 drivers/vfio/pci/vfio_pci_priv.h   |  1 -
 3 files changed, 43 insertions(+), 18 deletions(-)

diff --git a/drivers/vfio/pci/vfio_pci_config.c b/drivers/vfio/pci/vfio_pci_config.c
index 537e7f8bc884..014f865c1eaf 100644
--- a/drivers/vfio/pci/vfio_pci_config.c
+++ b/drivers/vfio/pci/vfio_pci_config.c
@@ -755,11 +755,15 @@ static bool vfio_af_flr_requested(int offset, __le32 val)
  * It takes all the required locks to protect the access of power related
  * variables and then invokes vfio_pci_set_power_state().
  */
-static void vfio_lock_and_set_power_state(struct vfio_pci_core_device *vdev,
+static int vfio_lock_and_set_power_state(struct vfio_pci_core_device *vdev,
 					  pci_power_t state)
 {
+	int ret;
+
 	if (state >= PCI_D3hot) {
-		vfio_pci_zap_and_down_write_memory_lock(vdev);
+		ret = vfio_pci_begin_bar_state_change(vdev);
+		if (ret)
+			return ret;
 		vfio_pci_dma_buf_move(vdev, true);
 	} else {
 		down_write(&vdev->memory_lock);
@@ -768,19 +772,34 @@ static void vfio_lock_and_set_power_state(struct vfio_pci_core_device *vdev,
 	vfio_pci_set_power_state(vdev, state);
 	if (__vfio_pci_memory_enabled(vdev))
 		vfio_pci_dma_buf_move(vdev, false);
-	up_write(&vdev->memory_lock);
+	if (state >= PCI_D3hot)
+		vfio_pci_end_bar_state_change(vdev);
+	else
+		up_write(&vdev->memory_lock);
+	return 0;
+}
+
+static bool vfio_pm_low_power_requested(int offset, __le32 val)
+{
+	return offset == PCI_PM_CTRL &&
+	       (le32_to_cpu(val) & PCI_PM_CTRL_STATE_MASK);
 }
 
 static int vfio_pm_config_write(struct vfio_pci_core_device *vdev, int pos,
 				int count, struct perm_bits *perm,
 				int offset, __le32 val)
 {
+	if (vfio_pm_low_power_requested(offset, val) &&
+	    !vfio_pci_bar_state_change_allowed(vdev))
+		return -EBUSY;
+
 	count = vfio_default_config_write(vdev, pos, count, perm, offset, val);
 	if (count < 0)
 		return count;
 
 	if (offset == PCI_PM_CTRL) {
 		pci_power_t state;
+		int ret;
 
 		switch (le32_to_cpu(val) & PCI_PM_CTRL_STATE_MASK) {
 		case 0:
@@ -797,7 +816,10 @@ static int vfio_pm_config_write(struct vfio_pci_core_device *vdev, int pos,
 			break;
 		}
 
-		vfio_lock_and_set_power_state(vdev, state);
+		ret = vfio_lock_and_set_power_state(vdev, state);
+
+		if (ret)
+			return ret;
 	}
 
 	return count;
diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index f31fbd980f3e..6612728a21a4 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -373,24 +373,30 @@ int vfio_pci_set_power_state(struct vfio_pci_core_device *vdev, pci_power_t stat
 static int vfio_pci_runtime_pm_entry(struct vfio_pci_core_device *vdev,
 				     struct eventfd_ctx *efdctx)
 {
+	int ret;
+
 	/*
 	 * The vdev power related flags are protected with 'memory_lock'
 	 * semaphore.
 	 */
-	vfio_pci_zap_and_down_write_memory_lock(vdev);
+	ret = vfio_pci_begin_bar_state_change(vdev);
+	if (ret)
+		return ret;
 	vfio_pci_dma_buf_move(vdev, true);
 
 	if (vdev->pm_runtime_engaged) {
-		up_write(&vdev->memory_lock);
-		return -EINVAL;
+		ret = -EINVAL;
+		goto out_unlock;
 	}
 
-	vdev->pm_runtime_engaged = true;
+	WRITE_ONCE(vdev->pm_runtime_engaged, true);
 	vdev->pm_wake_eventfd_ctx = efdctx;
 	pm_runtime_put_noidle(&vdev->pdev->dev);
-	up_write(&vdev->memory_lock);
+	ret = 0;
 
-	return 0;
+out_unlock:
+	vfio_pci_end_bar_state_change(vdev);
+	return ret;
 }
 
 static int vfio_pci_core_pm_entry(struct vfio_pci_core_device *vdev, u32 flags,
@@ -445,7 +451,11 @@ static int vfio_pci_core_pm_entry_with_wakeup(
 static void __vfio_pci_runtime_pm_exit(struct vfio_pci_core_device *vdev)
 {
 	if (vdev->pm_runtime_engaged) {
-		vdev->pm_runtime_engaged = false;
+		/*
+		 * Guest_memfd provider setup reads this flag with READ_ONCE()
+		 * without memory_lock; runtime resume can clear it concurrently.
+		 */
+		WRITE_ONCE(vdev->pm_runtime_engaged, false);
 		pm_runtime_get_noresume(&vdev->pdev->dev);
 
 		if (vdev->pm_wake_eventfd_ctx) {
@@ -1705,12 +1715,6 @@ static void vfio_pci_zap_bars(struct vfio_pci_core_device *vdev)
 	unmap_mapping_range(core_vdev->inode->i_mapping, start, len, true);
 }
 
-void vfio_pci_zap_and_down_write_memory_lock(struct vfio_pci_core_device *vdev)
-{
-	down_write(&vdev->memory_lock);
-	vfio_pci_zap_bars(vdev);
-}
-
 int vfio_pci_begin_bar_state_change(struct vfio_pci_core_device *vdev)
 {
 	if (!vfio_pci_bar_state_change_allowed(vdev))
diff --git a/drivers/vfio/pci/vfio_pci_priv.h b/drivers/vfio/pci/vfio_pci_priv.h
index 0af93f6d9a8a..f7b3178cc890 100644
--- a/drivers/vfio/pci/vfio_pci_priv.h
+++ b/drivers/vfio/pci/vfio_pci_priv.h
@@ -75,7 +75,6 @@ void vfio_config_free(struct vfio_pci_core_device *vdev);
 int vfio_pci_set_power_state(struct vfio_pci_core_device *vdev,
 			     pci_power_t state);
 
-void vfio_pci_zap_and_down_write_memory_lock(struct vfio_pci_core_device *vdev);
 /* Success retains memory_lock through the hardware operation. */
 int vfio_pci_begin_bar_state_change(struct vfio_pci_core_device *vdev);
 void vfio_pci_end_bar_state_change(struct vfio_pci_core_device *vdev);
-- 
2.43.0

[RFC PATCH v4 18/25] vfio/pci: Track userspace BAR mappings

[Aneesh Kumar K.V (Arm)]

Track userspace mappings of PCI BARs so a later patch can reject
guest_memfd attachment while such mappings exist.

Count the initial mmap explicitly because vm_ops->open() is not called
for it. Use the VMA open and close callbacks to track duplication and
unmapping, keeping the count nonzero until the last mapping is unmapped.

Cc: Alex Williamson <alex@shazbot.org>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 drivers/vfio/pci/vfio_pci_core.c | 19 +++++++++++++++++++
 include/linux/vfio_pci_core.h    |  2 ++
 2 files changed, 21 insertions(+)

diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index 6612728a21a4..d4f9ce6b5dc6 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -1814,7 +1814,23 @@ static vm_fault_t vfio_pci_mmap_page_fault(struct vm_fault *vmf)
 	return vfio_pci_mmap_huge_fault(vmf, 0);
 }
 
+static void vfio_pci_mmap_open(struct vm_area_struct *vma)
+{
+	struct vfio_pci_core_device *vdev = vma->vm_private_data;
+
+	atomic_inc(&vdev->mmap_count);
+}
+
+static void vfio_pci_mmap_close(struct vm_area_struct *vma)
+{
+	struct vfio_pci_core_device *vdev = vma->vm_private_data;
+
+	atomic_dec(&vdev->mmap_count);
+}
+
 static const struct vm_operations_struct vfio_pci_mmap_ops = {
+	.open = vfio_pci_mmap_open,
+	.close = vfio_pci_mmap_close,
 	.fault = vfio_pci_mmap_page_fault,
 #ifdef CONFIG_ARCH_SUPPORTS_HUGE_PFNMAP
 	.huge_fault = vfio_pci_mmap_huge_fault,
@@ -1896,6 +1912,8 @@ int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct vm_area_struct *vma
 	vm_flags_set(vma, VM_ALLOW_ANY_UNCACHED | VM_IO | VM_PFNMAP |
 			VM_DONTEXPAND | VM_DONTDUMP);
 	vma->vm_ops = &vfio_pci_mmap_ops;
+	/* Initial mmap does not invoke vm_ops->open(). */
+	vfio_pci_mmap_open(vma);
 
 	return 0;
 }
@@ -2205,6 +2223,7 @@ int vfio_pci_core_init_dev(struct vfio_device *core_vdev)
 		return ret;
 	INIT_LIST_HEAD(&vdev->dmabufs);
 	init_rwsem(&vdev->memory_lock);
+	atomic_set(&vdev->mmap_count, 0);
 	xa_init(&vdev->ctx);
 
 	return 0;
diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index 2f3fa2787737..6001fda0e001 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -8,6 +8,7 @@
  * Author: Tom Lyon, pugs@cisco.com
  */
 
+#include <linux/atomic.h>
 #include <linux/device.h>
 #include <linux/mutex.h>
 #include <linux/pci.h>
@@ -150,6 +151,7 @@ struct vfio_pci_core_device {
 	struct vfio_pci_core_device	*sriov_pf_core_dev;
 	struct notifier_block	nb;
 	struct rw_semaphore	memory_lock;
+	atomic_t		mmap_count;
 	struct list_head	dmabufs;
 };
 
-- 
2.43.0

[RFC PATCH v4 19/25] vfio/pci: Restrict host BAR access for guest_memfd providers

[Aneesh Kumar K.V (Arm)]

Exclude direct VFIO BAR access for the lifetime of a guest_memfd
provider. Access through the provider is checked and revoked by
guest_memfd; the following VFIO paths would bypass that protection:

* PCI read/write dispatch can reach MMIO BARs directly, I/O-port BARs
  through port I/O, and ROM, VGA or driver-specific region accessors.
  Reject all non-config read/write with EBUSY at dispatch, rather than
  relying on low-level MMIO checks which do not cover I/O-port BARs.

* An ioeventfd registration installs a kernel handler that writes its
  configured BAR address when signalled. Deny new registrations with
  EBUSY because those writes do not fault through guest_memfd or check
  private attributes. Preserve removal so an existing handler can be
  drained before attachment.

* A BAR dma-buf export supplies PFNs to another kernel/DMA user outside
  guest_memfd revocation. Deny export with EBUSY while preserving
  feature probes; attachment also rejects existing exports.

ROM region discovery is itself a hardware access: pci_map_rom()
enables/maps the ROM and reads it to validate its size. Report no ROM
access or size while attached rather than running that path. Other
region discovery returns metadata without authorizing read/write or
mmap. Require custom capability discovery callbacks to avoid BAR access
while attached.

Allow IRQ configuration, including MSI-X SET_IRQS. The PCI core may
program, mask or tear down the hardware MSI-X table. Userspace must keep
the table pages shared while host MSI-X is active and reject private
table/PBA mappings when unsupported; the kernel does not enforce that
sharing policy. Generic ioeventfd BAR writes remain excluded
independently.

Generic VFIO excludes mmap. Existing VMAs and persistent BAR users
prevent provider attachment, so low-level BAR helpers need no additional
guest_memfd admission checks.

Cc: Alex Williamson <alex@shazbot.org>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Cc: David Hildenbrand <david@kernel.org>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 drivers/vfio/pci/vfio_pci_core.c   | 15 +++++++++++++++
 drivers/vfio/pci/vfio_pci_dmabuf.c |  4 ++++
 drivers/vfio/pci/vfio_pci_priv.h   |  6 ++++++
 drivers/vfio/pci/vfio_pci_rdwr.c   |  4 ++++
 include/linux/vfio_pci_core.h      |  1 +
 5 files changed, 30 insertions(+)

diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index d4f9ce6b5dc6..2789bfab4415 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -1169,6 +1169,10 @@ int vfio_pci_ioctl_get_region_info(struct vfio_device *core_vdev,
 		info->flags = 0;
 		info->size = 0;
 
+		/* Discovering ROM size maps and reads the ROM. */
+		if (!vfio_pci_bar_access_allowed(vdev))
+			break;
+
 		if (pci_resource_start(pdev, PCI_ROM_RESOURCE)) {
 			/*
 			 * Check ROM content is valid. Need to enable memory
@@ -1298,6 +1302,12 @@ static int vfio_pci_ioctl_set_irqs(struct vfio_pci_core_device *vdev,
 			return PTR_ERR(data);
 	}
 
+	/*
+	 * VFIO_DEVICE_SET_IRQS remains allowed while guest_memfd uses the device.
+	 * MSI-X configuration can access the hardware table through the PCI core,
+	 * bypassing guest_memfd access checks. Userspace must keep the table pages
+	 * shared while host MSI-X is in use.
+	 */
 	mutex_lock(&vdev->igate);
 
 	ret = vfio_pci_set_irqs_ioctl(vdev, hdr.flags, hdr.index, hdr.start,
@@ -1638,6 +1648,11 @@ static ssize_t vfio_pci_rw(struct vfio_pci_core_device *vdev, char __user *buf,
 	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
 	int ret;
 
+	/* Non-config-space access needs the BAR access check. */
+	if (index != VFIO_PCI_CONFIG_REGION_INDEX &&
+	    !vfio_pci_bar_access_allowed(vdev))
+		return -EBUSY;
+
 	if (index >= VFIO_PCI_NUM_REGIONS + vdev->num_regions)
 		return -EINVAL;
 
diff --git a/drivers/vfio/pci/vfio_pci_dmabuf.c b/drivers/vfio/pci/vfio_pci_dmabuf.c
index c16f460c01d6..8cec6c93c049 100644
--- a/drivers/vfio/pci/vfio_pci_dmabuf.c
+++ b/drivers/vfio/pci/vfio_pci_dmabuf.c
@@ -237,6 +237,10 @@ int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
 	if (ret != 1)
 		return ret;
 
+	/* The dispatch gmem_lock excludes attachment through export completion. */
+	if (!vfio_pci_bar_access_allowed(vdev))
+		return -EBUSY;
+
 	if (copy_from_user(&get_dma_buf, arg, sizeof(get_dma_buf)))
 		return -EFAULT;
 
diff --git a/drivers/vfio/pci/vfio_pci_priv.h b/drivers/vfio/pci/vfio_pci_priv.h
index f7b3178cc890..a2501759e805 100644
--- a/drivers/vfio/pci/vfio_pci_priv.h
+++ b/drivers/vfio/pci/vfio_pci_priv.h
@@ -18,6 +18,12 @@ vfio_pci_bar_state_change_allowed(struct vfio_pci_core_device *vdev)
 	return true;
 }
 
+static inline bool
+vfio_pci_bar_access_allowed(struct vfio_pci_core_device *vdev)
+{
+	return true;
+}
+
 struct vfio_pci_ioeventfd {
 	struct list_head	next;
 	struct vfio_pci_core_device	*vdev;
diff --git a/drivers/vfio/pci/vfio_pci_rdwr.c b/drivers/vfio/pci/vfio_pci_rdwr.c
index df8848087681..80392392b2db 100644
--- a/drivers/vfio/pci/vfio_pci_rdwr.c
+++ b/drivers/vfio/pci/vfio_pci_rdwr.c
@@ -410,6 +410,10 @@ int vfio_pci_ioeventfd(struct vfio_pci_core_device *vdev, loff_t offset,
 	struct vfio_pci_ioeventfd *ioeventfd;
 	void __iomem *io;
 
+	/* Removal is allowed, but new handlers would bypass guest_memfd. */
+	if (fd >= 0 && !vfio_pci_bar_access_allowed(vdev))
+		return -EBUSY;
+
 	/* Only support ioeventfds into BARs */
 	if (bar > VFIO_PCI_BAR5_REGION_INDEX)
 		return -EINVAL;
diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index 6001fda0e001..4ebd29069034 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -46,6 +46,7 @@ struct vfio_pci_regops {
 	int	(*mmap)(struct vfio_pci_core_device *vdev,
 			struct vfio_pci_region *region,
 			struct vm_area_struct *vma);
+	/* Must not access BAR memory while guest_memfd uses the device. */
 	int	(*add_capability)(struct vfio_pci_core_device *vdev,
 				  struct vfio_pci_region *region,
 				  struct vfio_info_cap *caps);
-- 
2.43.0

[RFC PATCH v4 20/25] vfio: Dispatch guest_memfd provider operations through the device file

[Aneesh Kumar K.V (Arm)]

Forward guest_memfd provider operations and the supplied device context
through the VFIO device file to the device driver. Validate the driver
callbacks before attachment. Keep a reference to the VFIO device file
until the driver's provider release callback returns, so the device and
driver operations remain available throughout cleanup.

Reject non-VFIO resource files and providers without device PFNs, whole-
range validation, mapping exclusion and release support. BAR
addresses are device PFNs rather than allocatable RAM folios; accepting
folio callbacks or incomplete exclusion callbacks would allow a backing
type that cannot honor device mapping and revocation rules. A driver
without this contract returns EOPNOTSUPP at attachment rather than
exposing its regions through a partial provider.

Cc: Alex Williamson <alex@shazbot.org>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 drivers/vfio/vfio_main.c | 108 +++++++++++++++++++++++++++++++++++++++
 include/linux/vfio.h     |   4 ++
 2 files changed, 112 insertions(+)

diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
index ff49f2fdb602..838242f82fd2 100644
--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -14,6 +14,7 @@
 #include <linux/compat.h>
 #include <linux/device.h>
 #include <linux/fs.h>
+#include <linux/guest_memfd.h>
 #include <linux/idr.h>
 #include <linux/iommu.h>
 #if IS_ENABLED(CONFIG_KVM)
@@ -248,6 +249,110 @@ struct vfio_device *_vfio_alloc_device(size_t size, struct device *dev,
 }
 EXPORT_SYMBOL_GPL(_vfio_alloc_device);
 
+#ifdef CONFIG_KVM_GUEST_MEMFD
+struct vfio_gmem_provider {
+	const struct guest_memfd_provider_operations *ops;
+	void *data;
+	struct file *file;
+};
+
+static void *vfio_gmem_provider_attach(struct inode *inode, struct file *file,
+				       struct guest_memfd_device *gmem)
+{
+	const struct guest_memfd_provider_operations *ops;
+	struct vfio_gmem_provider *provider;
+	struct vfio_device_file *df;
+	void *data;
+
+	/* Reject other file types before interpreting their private_data. */
+	if (file->f_op != &vfio_device_fops || !gmem)
+		return ERR_PTR(-EOPNOTSUPP);
+
+	df = file->private_data;
+	ops = df->device->ops->gmem_provider_ops;
+	/* Every provider must support attachment and release. */
+	if (!ops || !ops->attach || !ops->release)
+		return ERR_PTR(-EOPNOTSUPP);
+
+	/* VFIO requires device PFNs, range validation and mapping exclusion. */
+	if (!ops->get_pfn || !ops->device_memory ||
+	    !ops->prepare_memory_region ||
+	    !ops->lock ||
+	    !ops->unlock)
+		return ERR_PTR(-EOPNOTSUPP);
+
+	/* VFIO provides PFNs, not folios. */
+	if (ops->alloc_folio || ops->invalidate_folio)
+		return ERR_PTR(-EOPNOTSUPP);
+
+	provider = kzalloc_obj(*provider);
+	if (!provider)
+		return ERR_PTR(-ENOMEM);
+
+	data = ops->attach(inode, file, gmem);
+	if (IS_ERR(data)) {
+		kfree(provider);
+		return data;
+	}
+
+	provider->ops = ops;
+	provider->data = data;
+	/* Pin the driver operations through release(), including its return. */
+	provider->file = get_file(file);
+	return provider;
+}
+
+static void vfio_gmem_provider_release(void *data)
+{
+	struct vfio_gmem_provider *provider = data;
+
+	provider->ops->release(provider->data);
+	fput(provider->file);
+	kfree(provider);
+}
+
+static int vfio_gmem_provider_prepare_memory_region(void *data,
+		u64 offset, u64 size, u32 flags)
+{
+	struct vfio_gmem_provider *provider = data;
+
+	return provider->ops->prepare_memory_region(provider->data, offset,
+						    size, flags);
+}
+
+static int vfio_gmem_provider_get_pfn(void *data, pgoff_t index,
+		unsigned long *pfn, int *max_order)
+{
+	struct vfio_gmem_provider *provider = data;
+
+	return provider->ops->get_pfn(provider->data, index, pfn, max_order);
+}
+
+static void vfio_gmem_provider_lock(void *context)
+{
+	struct vfio_gmem_provider *provider = context;
+
+	provider->ops->lock(provider->data);
+}
+
+static void vfio_gmem_provider_unlock(void *context)
+{
+	struct vfio_gmem_provider *provider = context;
+
+	provider->ops->unlock(provider->data);
+}
+
+static const struct guest_memfd_provider_operations vfio_gmem_provider_ops = {
+	.attach = vfio_gmem_provider_attach,
+	.release = vfio_gmem_provider_release,
+	.prepare_memory_region = vfio_gmem_provider_prepare_memory_region,
+	.lock = vfio_gmem_provider_lock,
+	.unlock = vfio_gmem_provider_unlock,
+	.get_pfn = vfio_gmem_provider_get_pfn,
+	.device_memory = true,
+};
+#endif
+
 static int vfio_fs_init_fs_context(struct fs_context *fc)
 {
 	return init_pseudo(fc, VFIO_MAGIC) ? 0 : -ENOMEM;
@@ -1449,6 +1554,9 @@ static void vfio_device_show_fdinfo(struct seq_file *m, struct file *filep)
 #endif
 
 const struct file_operations vfio_device_fops = {
+#ifdef CONFIG_KVM_GUEST_MEMFD
+	.gmem_provider_ops = &vfio_gmem_provider_ops,
+#endif
 	.owner		= THIS_MODULE,
 	.open		= vfio_device_fops_cdev_open,
 	.release	= vfio_device_fops_release,
diff --git a/include/linux/vfio.h b/include/linux/vfio.h
index 2436b8bd600a..10fd5015ce1d 100644
--- a/include/linux/vfio.h
+++ b/include/linux/vfio.h
@@ -88,6 +88,8 @@ struct vfio_device {
 	struct dentry *debug_root;
 };
 
+struct guest_memfd_provider_operations;
+
 /**
  * struct vfio_device_ops - VFIO bus driver device callbacks
  *
@@ -118,9 +120,11 @@ struct vfio_device {
  *         if none was provided.
  * @dma_unmap: Called when userspace unmaps IOVA from the container
  *             this device is attached to.
+ * @gmem_provider_ops: Optional driver guest_memfd resource operations
  * @device_feature: Optional, fill in the VFIO_DEVICE_FEATURE ioctl
  */
 struct vfio_device_ops {
+	const struct guest_memfd_provider_operations *gmem_provider_ops;
 	char	*name;
 	int	(*init)(struct vfio_device *vdev);
 	void	(*release)(struct vfio_device *vdev);
-- 
2.43.0

[RFC PATCH v4 21/25] vfio: Serialize guest_memfd attachment with device file operations

[Aneesh Kumar K.V (Arm)]

Attaching a VFIO device to guest_memfd must not race with VFIO file
operations that access its BARs or create mappings. The attachment state
must apply to every open VFIO file referring to the same device.

Hold the device's gmem_lock for reading across read, write, mmap and
normal ioctl dispatch, and for writing across provider attachment and
release. File operations already in progress finish before attachment
proceeds. Later operations observe the updated gmem_attached state while
holding the same lock, allowing driver checks to reject BAR access.

Hold dev_set->lock across the driver attach/release callbacks and updates
to gmem_attached as well. This lets hot reset check every affected device
for an attachment under the device-set lock without racing attachment or
release. Driver callbacks run with both locks held and must not reacquire
them.

Return EBUSY for VFIO mmap while guest_memfd is attached, including mmap
through another open VFIO file. Such a mapping would insert BAR PFNs
without checking guest_memfd attributes

Keep read, write and ioctl dispatch available to the driver. The PCI
checks reject operations that access BARs or disrupt their mappings while
preserving configuration queries, unrelated ioctls, feature probes and
cleanup.

Cc: Alex Williamson <alex@shazbot.org>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 drivers/vfio/pci/vfio_pci_config.c |  2 ++
 drivers/vfio/pci/vfio_pci_core.c   |  3 +++
 drivers/vfio/vfio_main.c           | 28 +++++++++++++++++++++++++++-
 include/linux/vfio.h               |  4 +++-
 4 files changed, 35 insertions(+), 2 deletions(-)

diff --git a/drivers/vfio/pci/vfio_pci_config.c b/drivers/vfio/pci/vfio_pci_config.c
index 014f865c1eaf..2cd855cc50bc 100644
--- a/drivers/vfio/pci/vfio_pci_config.c
+++ b/drivers/vfio/pci/vfio_pci_config.c
@@ -599,6 +599,8 @@ static int vfio_basic_config_write(struct vfio_pci_core_device *vdev, int pos,
 		}
 
 		/*
+		 * vfio_device_fops_write() holds gmem_lock for read, keeping
+		 * gmem_attached unchanged across this write.
 		 * Check BME and Memory Enable changes while holding memory_lock
 		 * for write, before updating hardware or vconfig. Return an
 		 * error rather than silently masking a rejected write.
diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index 2789bfab4415..b8e574194f07 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -378,6 +378,9 @@ static int vfio_pci_runtime_pm_entry(struct vfio_pci_core_device *vdev,
 	/*
 	 * The vdev power related flags are protected with 'memory_lock'
 	 * semaphore.
+	 *
+	 * vfio_device_fops_unl_ioctl() holds gmem_lock for read,
+	 * preventing provider setup during this operation.
 	 */
 	ret = vfio_pci_begin_bar_state_change(vdev);
 	if (ret)
diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
index 838242f82fd2..787d9a661f75 100644
--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -289,12 +289,21 @@ static void *vfio_gmem_provider_attach(struct inode *inode, struct file *file,
 	if (!provider)
 		return ERR_PTR(-ENOMEM);
 
+	guard(rwsem_write)(&df->device->gmem_lock);
+	if (df->device->gmem_attached) {
+		kfree(provider);
+		return ERR_PTR(-EBUSY);
+	}
+
+	/* Publish attachment under both locks for device-set reset admission. */
+	guard(mutex)(&df->device->dev_set->lock);
 	data = ops->attach(inode, file, gmem);
 	if (IS_ERR(data)) {
 		kfree(provider);
 		return data;
 	}
 
+	df->device->gmem_attached = true;
 	provider->ops = ops;
 	provider->data = data;
 	/* Pin the driver operations through release(), including its return. */
@@ -306,7 +315,13 @@ static void vfio_gmem_provider_release(void *data)
 {
 	struct vfio_gmem_provider *provider = data;
 
-	provider->ops->release(provider->data);
+	struct vfio_device_file *df = provider->file->private_data;
+
+	scoped_guard(rwsem_write, &df->device->gmem_lock) {
+		guard(mutex)(&df->device->dev_set->lock);
+		provider->ops->release(provider->data);
+		df->device->gmem_attached = false;
+	}
 	fput(provider->file);
 	kfree(provider);
 }
@@ -1447,6 +1462,8 @@ static long vfio_device_fops_unl_ioctl(struct file *filep,
 	if (cmd == VFIO_DEVICE_BIND_IOMMUFD)
 		return vfio_df_ioctl_bind_iommufd(df, uptr);
 
+	guard(rwsem_read)(&device->gmem_lock);
+
 	/* Paired with smp_store_release() following vfio_df_open() */
 	if (!smp_load_acquire(&df->access_granted))
 		return -EINVAL;
@@ -1495,6 +1512,8 @@ static ssize_t vfio_device_fops_read(struct file *filep, char __user *buf,
 	struct vfio_device_file *df = filep->private_data;
 	struct vfio_device *device = df->device;
 
+	guard(rwsem_read)(&device->gmem_lock);
+
 	/* Paired with smp_store_release() following vfio_df_open() */
 	if (!smp_load_acquire(&df->access_granted))
 		return -EINVAL;
@@ -1512,6 +1531,8 @@ static ssize_t vfio_device_fops_write(struct file *filep,
 	struct vfio_device_file *df = filep->private_data;
 	struct vfio_device *device = df->device;
 
+	guard(rwsem_read)(&device->gmem_lock);
+
 	/* Paired with smp_store_release() following vfio_df_open() */
 	if (!smp_load_acquire(&df->access_granted))
 		return -EINVAL;
@@ -1527,10 +1548,15 @@ static int vfio_device_fops_mmap(struct file *filep, struct vm_area_struct *vma)
 	struct vfio_device_file *df = filep->private_data;
 	struct vfio_device *device = df->device;
 
+	guard(rwsem_read)(&device->gmem_lock);
+
 	/* Paired with smp_store_release() following vfio_df_open() */
 	if (!smp_load_acquire(&df->access_granted))
 		return -EINVAL;
 
+	if (device->gmem_attached)
+		return -EBUSY;
+
 	if (unlikely(!device->ops->mmap))
 		return -EINVAL;
 
diff --git a/include/linux/vfio.h b/include/linux/vfio.h
index 10fd5015ce1d..ceab68b76300 100644
--- a/include/linux/vfio.h
+++ b/include/linux/vfio.h
@@ -120,7 +120,9 @@ struct guest_memfd_provider_operations;
  *         if none was provided.
  * @dma_unmap: Called when userspace unmaps IOVA from the container
  *             this device is attached to.
- * @gmem_provider_ops: Optional driver guest_memfd resource operations
+ * @gmem_provider_ops: Optional driver guest_memfd resource operations. VFIO
+ *                     calls attach/release with gmem_lock held for write and
+ *                     dev_set->lock held; callbacks must not reacquire them.
  * @device_feature: Optional, fill in the VFIO_DEVICE_FEATURE ioctl
  */
 struct vfio_device_ops {
-- 
2.43.0

[RFC PATCH v4 22/25] vfio/pci: Require D0 before providing guest_memfd BAR backing

[Aneesh Kumar K.V (Arm)]

Guest_memfd needs the device in D0 while its BARs supply backing memory.
Lower-power states can limit device operation or make BAR accesses
unavailable; D3cold removes power from the device entirely.

Add vfio_pci_core_pm_pin_d0() to resume the device and retain a
runtime-PM reference. Check current_state under memory_lock and return
EBUSY unless the device is in D0. An explicit PMCSR write can leave the
device in a lower power state even when runtime PM considers it active.

The retained reference prevents runtime suspend. The caller must also
prevent explicit power-state changes and drop the reference with
pm_runtime_put() when finished. The following provider patch holds it
until provider release.

Cc: Alex Williamson <alex@shazbot.org>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 drivers/vfio/pci/vfio_pci_core.c | 28 ++++++++++++++++++++++++++++
 drivers/vfio/pci/vfio_pci_priv.h |  2 ++
 2 files changed, 30 insertions(+)

diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index b8e574194f07..1c606da81879 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -370,6 +370,34 @@ int vfio_pci_set_power_state(struct vfio_pci_core_device *vdev, pci_power_t stat
 	return ret;
 }
 
+/**
+ * vfio_pci_core_pm_pin_d0() - resume a device and retain runtime PM
+ * @vdev: device whose runtime-PM reference is acquired
+ *
+ * Require D0 after resuming: explicit PMCSR writes can leave a
+ * runtime-active function in a lower power state. The caller must exclude
+ * explicit power-state changes while relying on this reference.
+ *
+ * Return: 0 with a reference to release using pm_runtime_put(), or a negative
+ * error with no reference held.
+ */
+int vfio_pci_core_pm_pin_d0(struct vfio_pci_core_device *vdev)
+{
+	int ret;
+
+	ret = pm_runtime_resume_and_get(&vdev->pdev->dev);
+	if (ret < 0)
+		return ret;
+
+	down_read(&vdev->memory_lock);
+	ret = vdev->pdev->current_state == PCI_D0 ? 0 : -EBUSY;
+	up_read(&vdev->memory_lock);
+	if (ret)
+		pm_runtime_put(&vdev->pdev->dev);
+
+	return ret;
+}
+
 static int vfio_pci_runtime_pm_entry(struct vfio_pci_core_device *vdev,
 				     struct eventfd_ctx *efdctx)
 {
diff --git a/drivers/vfio/pci/vfio_pci_priv.h b/drivers/vfio/pci/vfio_pci_priv.h
index a2501759e805..43d74c46e841 100644
--- a/drivers/vfio/pci/vfio_pci_priv.h
+++ b/drivers/vfio/pci/vfio_pci_priv.h
@@ -78,6 +78,8 @@ void vfio_pci_uninit_perm_bits(void);
 int vfio_config_init(struct vfio_pci_core_device *vdev);
 void vfio_config_free(struct vfio_pci_core_device *vdev);
 
+int vfio_pci_core_pm_pin_d0(struct vfio_pci_core_device *vdev);
+
 int vfio_pci_set_power_state(struct vfio_pci_core_device *vdev,
 			     pci_power_t state);
 
-- 
2.43.0

[RFC PATCH v4 23/25] iommufd: Support MMIO provider attachment to vDEVICEs

[Aneesh Kumar K.V (Arm)]

Allow a device MMIO provider to attach to the existing vDEVICE after the
backend validates its association with the VM. Keep the vDEVICE and
IOMMUFD context alive until the provider detaches.

Add a per-vDEVICE MMIO mutex to serialize provider attachment and
detachment, and allow providers and backends to use the same lock for
memory conversion and device state changes. Provide a callback to query
whether private MMIO ranges remain.

Cc: Kevin Tian <kevin.tian@intel.com>
Cc: "Joerg Roedel (AMD)" <joro@8bytes.org>
Cc: Robin Murphy <robin.murphy@arm.com>
Cc: iommu@lists.linux.dev
Cc: patches@lists.linux.dev
Cc: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 drivers/iommu/iommufd/viommu.c | 86 ++++++++++++++++++++++++++++++++++
 include/linux/iommufd.h        | 26 ++++++++++
 2 files changed, 112 insertions(+)

diff --git a/drivers/iommu/iommufd/viommu.c b/drivers/iommu/iommufd/viommu.c
index 6479083be095..09dedd4ece6d 100644
--- a/drivers/iommu/iommufd/viommu.c
+++ b/drivers/iommu/iommufd/viommu.c
@@ -290,6 +290,7 @@ int iommufd_vdevice_alloc_ioctl(struct iommufd_ucmd *ucmd)
 		goto out_unlock_igroup;
 	}
 
+	mutex_init(&vdev->mmio_lock);
 	vdev->virt_id = virt_id;
 	vdev->viommu = viommu;
 	refcount_inc(&viommu->obj.users);
@@ -527,3 +528,88 @@ int iommufd_hw_queue_alloc_ioctl(struct iommufd_ucmd *ucmd)
 	iommufd_put_object(ucmd->ictx, &viommu->obj);
 	return rc;
 }
+
+struct iommufd_vdevice *
+iommufd_device_attach_mmio_provider(struct iommufd_device *idev,
+		struct kvm *kvm, void *data,
+		bool (*has_private_mmio)(void *),
+		unsigned long *owner)
+{
+	struct iommufd_vdevice *vdev;
+	int ret;
+
+	if (!idev)
+		return ERR_PTR(-ENODEV);
+
+	guard(mutex)(&idev->igroup->lock);
+	if (idev->destroying)
+		return ERR_PTR(-EOPNOTSUPP);
+
+	vdev = idev->vdev;
+	if (!vdev)
+		return ERR_PTR(-EOPNOTSUPP);
+
+	if (!vdev->viommu->ops ||
+	    !vdev->viommu->ops->vdevice_get_mmio_owner)
+		return ERR_PTR(-EOPNOTSUPP);
+
+	guard(mutex)(&vdev->mmio_lock);
+	if (vdev->mmio_provider_data)
+		return ERR_PTR(-EBUSY);
+
+	ret = vdev->viommu->ops->vdevice_get_mmio_owner(vdev, kvm, owner);
+	if (ret)
+		return ERR_PTR(ret);
+
+	/* IOMMU_DESTROY must wait for the attached gmem provider to detach. */
+	ret = iommufd_try_inc_users(idev->ictx, &vdev->obj);
+	if (ret)
+		return ERR_PTR(ret);
+
+	iommufd_ctx_get(idev->ictx);
+	vdev->mmio_provider_data = data;
+	vdev->has_private_mmio = has_private_mmio;
+	return vdev;
+}
+EXPORT_SYMBOL_NS_GPL(iommufd_device_attach_mmio_provider, "IOMMUFD");
+
+void iommufd_vdevice_detach_mmio_provider(struct iommufd_vdevice *vdev)
+{
+	struct iommufd_ctx *ictx = vdev->viommu->ictx;
+
+	scoped_guard(mutex, &vdev->mmio_lock) {
+		WARN_ON(iommufd_vdevice_has_private_mmio(vdev));
+		vdev->mmio_provider_data = NULL;
+		vdev->has_private_mmio = NULL;
+	}
+	refcount_dec(&vdev->obj.users);
+	iommufd_ctx_put(ictx);
+}
+EXPORT_SYMBOL_NS_GPL(iommufd_vdevice_detach_mmio_provider, "IOMMUFD");
+
+void iommufd_vdevice_mmio_lock(struct iommufd_vdevice *vdev)
+{
+	mutex_lock(&vdev->mmio_lock);
+}
+EXPORT_SYMBOL_NS_GPL(iommufd_vdevice_mmio_lock, "IOMMUFD");
+
+void iommufd_vdevice_mmio_unlock(struct iommufd_vdevice *vdev)
+{
+	mutex_unlock(&vdev->mmio_lock);
+}
+EXPORT_SYMBOL_NS_GPL(iommufd_vdevice_mmio_unlock, "IOMMUFD");
+
+/**
+ * iommufd_vdevice_has_private_mmio() - check for private MMIO ranges
+ * @vdev: vDEVICE whose private MMIO state is being checked
+ *
+ * Return: true if the attached provider has committed private ranges.
+ */
+bool iommufd_vdevice_has_private_mmio(struct iommufd_vdevice *vdev)
+{
+	lockdep_assert_held(&vdev->mmio_lock);
+	if (!vdev->has_private_mmio)
+		return false;
+	return vdev->has_private_mmio(vdev->mmio_provider_data);
+}
+EXPORT_SYMBOL_NS_GPL(iommufd_vdevice_has_private_mmio, "IOMMUFD");
diff --git a/include/linux/iommufd.h b/include/linux/iommufd.h
index 4d5995956cce..0c57073b0f99 100644
--- a/include/linux/iommufd.h
+++ b/include/linux/iommufd.h
@@ -7,9 +7,11 @@
 #define __LINUX_IOMMUFD_H
 
 #include <linux/bits.h>
+#include <linux/cleanup.h>
 #include <linux/err.h>
 #include <linux/errno.h>
 #include <linux/iommu.h>
+#include <linux/mutex.h>
 #include <linux/refcount.h>
 #include <linux/types.h>
 #include <linux/xarray.h>
@@ -26,6 +28,8 @@ struct iommufd_device;
 struct iommufd_viommu_ops;
 struct tsm_dev;
 struct page;
+struct kvm;
+struct iommufd_vdevice;
 
 enum iommufd_object_type {
 	IOMMUFD_OBJ_NONE,
@@ -72,6 +76,19 @@ int iommufd_device_replace(struct iommufd_device *idev, ioasid_t pasid,
 			   u32 *pt_id);
 void iommufd_device_detach(struct iommufd_device *idev, ioasid_t pasid);
 
+/* The returned opaque vDEVICE pins its IOMMUFD context and physical owner. */
+struct iommufd_vdevice *
+iommufd_device_attach_mmio_provider(struct iommufd_device *idev,
+		struct kvm *kvm, void *data,
+		bool (*has_private_mmio)(void *),
+		unsigned long *owner);
+void iommufd_vdevice_detach_mmio_provider(struct iommufd_vdevice *vdev);
+void iommufd_vdevice_mmio_lock(struct iommufd_vdevice *vdev);
+void iommufd_vdevice_mmio_unlock(struct iommufd_vdevice *vdev);
+DEFINE_GUARD(iommufd_vdevice_mmio, struct iommufd_vdevice *,
+	     iommufd_vdevice_mmio_lock(_T), iommufd_vdevice_mmio_unlock(_T))
+bool iommufd_vdevice_has_private_mmio(struct iommufd_vdevice *vdev);
+
 struct iommufd_ctx *iommufd_device_to_ictx(struct iommufd_device *idev);
 u32 iommufd_device_to_id(struct iommufd_device *idev);
 
@@ -132,6 +149,11 @@ struct iommufd_vdevice {
 	/* Supported vDEVICE operations; set by vdevice_init(). */
 	u64 supported_ops;
 
+	/* Serialize attachment and conversion of device memory. */
+	struct mutex mmio_lock;
+	void *mmio_provider_data;
+	bool (*has_private_mmio)(void *data);
+
 	/* Clean up all driver-specific parts of an iommufd_vdevice */
 	void (*destroy)(struct iommufd_vdevice *vdev);
 };
@@ -179,6 +201,7 @@ struct iommufd_hw_queue {
  *                    which case the core re-invokes the op for the remainder.
  *                    The data structure of the array entry must be defined in
  *                    include/uapi/linux/iommufd.h
+ * @vdevice_get_mmio_owner: Validate the VM and return its MMIO owner.
  * @vdevice_size: Size of the driver-defined vDEVICE structure per this vIOMMU
  * @vdevice_init: Initialize the driver-level structure of a vDEVICE object, or
  *                related HW procedure. @vdev is already initialized by iommufd
@@ -217,6 +240,9 @@ struct iommufd_viommu_ops {
 		const struct iommu_user_data *user_data);
 	int (*cache_invalidate)(struct iommufd_viommu *viommu,
 				struct iommu_user_data_array *array);
+	/* Owner lookup runs with vdevice->mmio_lock held. */
+	int (*vdevice_get_mmio_owner)(struct iommufd_vdevice *vdev,
+		struct kvm *kvm, unsigned long *owner);
 	const size_t vdevice_size;
 	int (*vdevice_init)(struct iommufd_vdevice *vdev);
 	size_t (*get_hw_queue_size)(struct iommufd_viommu *viommu,
-- 
2.43.0

[RFC PATCH v4 24/25] vfio/pci: Add a guest_memfd resource provider for MMIO BARs

[Aneesh Kumar K.V (Arm)]

Allow a VFIO PCI device file to supply MMIO BAR PFNs to guest_memfd.
Validate BAR ranges before memslot setup and provide PFNs for shared
faults and private conversion.

Require page-aligned, nonempty ranges within the BAR and guest_memfd
file. Reject memslots requesting dirty logging or read-only access,
which this provider does not implement.

Existing VFIO mappings, dma-buf exports and ioeventfd handlers can
access BARs without guest_memfd attribute checks. Private conversion
cannot revoke those accesses, so reject setup with EBUSY while any
exist.

Require an IOMMUFD vDEVICE associated with the same VM as guest_memfd.
Keep that vDEVICE referenced until release and pass its device ID and
mapping-owner handle to guest_memfd for KVM device mappings.

Require D0 and hold a runtime-PM reference until release to keep BARs
accessible. Reject setup while userspace allows runtime suspend.
Block direct VFIO BAR access, reset, low-power entry and userspace
changes to Memory Enable or Bus Master Enable while the device
supplies BAR PFNs to guest_memfd.

Use the vDEVICE mmio_lock to prevent device unlock from racing with
conversion, invalidation and attribute updates. PFN lookup holds
memory_lock for read while checking BAR access.

MSI-X configuration remains allowed. Userspace must keep the table/PBA
pages shared while host MSI-X is active.

Cc: Alex Williamson <alex@shazbot.org>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: kvm@vger.kernel.org
Cc: patches@lists.linux.dev
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 drivers/vfio/pci/Kconfig         |  11 ++
 drivers/vfio/pci/Makefile        |   1 +
 drivers/vfio/pci/vfio_pci.c      |   3 +
 drivers/vfio/pci/vfio_pci_gmem.c | 327 +++++++++++++++++++++++++++++++
 drivers/vfio/pci/vfio_pci_gmem.h |  17 ++
 drivers/vfio/pci/vfio_pci_priv.h |  15 +-
 include/linux/vfio_pci_core.h    |   4 +
 7 files changed, 375 insertions(+), 3 deletions(-)
 create mode 100644 drivers/vfio/pci/vfio_pci_gmem.c
 create mode 100644 drivers/vfio/pci/vfio_pci_gmem.h

diff --git a/drivers/vfio/pci/Kconfig b/drivers/vfio/pci/Kconfig
index 296bf01e185e..e5dedf3e4a65 100644
--- a/drivers/vfio/pci/Kconfig
+++ b/drivers/vfio/pci/Kconfig
@@ -55,6 +55,17 @@ config VFIO_PCI_ZDEV_KVM
 
 	  To enable s390x KVM vfio-pci extensions, say Y.
 
+config VFIO_PCI_GMEM
+	bool "VFIO PCI device guest_memfd resource provider"
+	depends on VFIO_PCI && IOMMUFD && KVM
+	depends on KVM_GUEST_MEMFD
+	help
+	  Allow the VFIO PCI device fd to provide BAR PFNs to guest_memfd,
+	  including confidential-device range conversion through its
+	  existing IOMMUFD vDEVICE. This requires a capable TSM backend.
+	  Shared access is revoked before private range mapping and is
+	  restored only after private mappings and device state permit it.
+
 config VFIO_PCI_DMABUF
 	def_bool y if VFIO_PCI_CORE && PCI_P2PDMA && DMA_SHARED_BUFFER
 
diff --git a/drivers/vfio/pci/Makefile b/drivers/vfio/pci/Makefile
index 6138f1bf241d..351af00acb3f 100644
--- a/drivers/vfio/pci/Makefile
+++ b/drivers/vfio/pci/Makefile
@@ -3,6 +3,7 @@
 vfio-pci-core-y := vfio_pci_core.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o
 vfio-pci-core-$(CONFIG_VFIO_PCI_ZDEV_KVM) += vfio_pci_zdev.o
 vfio-pci-core-$(CONFIG_VFIO_PCI_DMABUF) += vfio_pci_dmabuf.o
+vfio-pci-core-$(CONFIG_VFIO_PCI_GMEM) += vfio_pci_gmem.o
 obj-$(CONFIG_VFIO_PCI_CORE) += vfio-pci-core.o
 
 vfio-pci-y := vfio_pci.o
diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c
index e7c012febe91..64068e7aa61e 100644
--- a/drivers/vfio/pci/vfio_pci.c
+++ b/drivers/vfio/pci/vfio_pci.c
@@ -147,6 +147,9 @@ static int vfio_pci_init_dev(struct vfio_device *core_vdev)
 }
 
 static const struct vfio_device_ops vfio_pci_ops = {
+#if IS_ENABLED(CONFIG_VFIO_PCI_GMEM)
+	.gmem_provider_ops = &vfio_pci_gmem_provider_ops,
+#endif
 	.name		= "vfio-pci",
 	.init		= vfio_pci_init_dev,
 	.release	= vfio_pci_core_release_dev,
diff --git a/drivers/vfio/pci/vfio_pci_gmem.c b/drivers/vfio/pci/vfio_pci_gmem.c
new file mode 100644
index 000000000000..11f6ae44cd49
--- /dev/null
+++ b/drivers/vfio/pci/vfio_pci_gmem.c
@@ -0,0 +1,327 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2026 ARM Ltd.
+ */
+#include <linux/guest_memfd.h>
+#include <linux/iommufd.h>
+#include <linux/kvm.h>
+#include <linux/mm.h>
+#include <linux/overflow.h>
+#include <linux/pci.h>
+#include <linux/pm_runtime.h>
+#include <linux/slab.h>
+#include <linux/vfio_pci_core.h>
+
+#include "../vfio.h"
+#include "vfio_pci_priv.h"
+#include "vfio_pci_gmem.h"
+
+/**
+ * vfio_gmem_resolve_bar_range() - validate and resolve a VFIO BAR interval
+ * @ctx: attached VFIO provider
+ * @offset: byte offset in the VFIO region namespace
+ * @size: interval length in bytes
+ * @bar: output BAR index
+ * @bar_offset: output byte offset within that BAR
+ * @pa: output starting physical address
+ *
+ * Return: 0 on success or -EINVAL for an ineligible interval.
+ */
+static int vfio_gmem_resolve_bar_range(struct vfio_pci_gmem *ctx, u64 offset,
+		u64 size, u32 *bar, u64 *bar_offset, phys_addr_t *pa)
+{
+	struct pci_dev *pdev = ctx->vdev->pdev;
+	resource_size_t start, len;
+	u32 index = offset >> VFIO_PCI_OFFSET_SHIFT;
+	u64 off = offset & ((1ULL << VFIO_PCI_OFFSET_SHIFT) - 1);
+
+	if (!size || !PAGE_ALIGNED(offset | size))
+		return -EINVAL;
+
+	if (offset >= ctx->gmem->size ||
+	    size > ctx->gmem->size - offset)
+		return -EINVAL;
+
+	if (index >= PCI_STD_NUM_BARS)
+		return -EINVAL;
+
+	start = pci_resource_start(pdev, index);
+	len = pci_resource_len(pdev, index);
+
+	if (!ctx->vdev->bar_mmap_supported[index] ||
+	    !(pci_resource_flags(pdev, index) & IORESOURCE_MEM) ||
+	    !PAGE_ALIGNED(start))
+		return -EINVAL;
+
+	if (off >= len || size > len - off)
+		return -EINVAL;
+
+	/* The interval must not cross into the next VFIO region. */
+	if (size > (1ULL << VFIO_PCI_OFFSET_SHIFT) - off)
+		return -EINVAL;
+
+	if (check_add_overflow(start, off, pa))
+		return -EINVAL;
+
+	*bar = index;
+	*bar_offset = off;
+	return 0;
+}
+
+/**
+ * vfio_gmem_has_private_mmio() - check for private BAR ranges
+ * @data: attached VFIO provider context
+ *
+ * Return: true if the core has committed private BAR ranges.
+ */
+static bool vfio_gmem_has_private_mmio(void *data)
+{
+	struct vfio_pci_gmem *ctx = data;
+
+	/* The caller holds mmio_lock, including before attach() returns. */
+	return READ_ONCE(ctx->gmem->has_private_ranges);
+}
+
+/**
+ * vfio_gmem_attach() - prepare a VFIO PCI device to supply guest_memfd backing
+ * @inode: guest_memfd inode supplied by the core
+ * @file: open VFIO device file supplying the BAR namespace
+ * @gdev: device memory context supplied by guest_memfd
+ *
+ * Require @file to be bound to IOMMUFD. Reject the request if VFIO BAR
+ * mappings, dma-buf exports or ioeventfd handlers already exist.
+ *
+ * Require the device to be in D0 and retain a runtime-PM reference to
+ * prevent runtime suspend. Obtain the existing IOMMUFD vDEVICE after
+ * checking that it belongs to the same VM as @gdev. Keep both references
+ * until vfio_gmem_release().
+ *
+ * Generic VFIO keeps @file referenced until vfio_gmem_release() returns.
+ *
+ * Return: Provider context on success, or an ERR_PTR() on failure.
+ */
+static void *vfio_gmem_attach(struct inode *inode, struct file *file,
+		struct guest_memfd_device *gdev)
+{
+	struct vfio_device_file *df = file->private_data;
+	struct vfio_pci_core_device *vdev =
+		container_of(df->device, struct vfio_pci_core_device, vdev);
+	struct vfio_pci_gmem *ctx;
+	struct iommufd_vdevice *ivdev;
+	int ret;
+
+	if (!gdev)
+		return ERR_PTR(-EOPNOTSUPP);
+
+	ctx = kzalloc_obj(*ctx);
+	if (!ctx)
+		return ERR_PTR(-ENOMEM);
+
+	ctx->vdev = vdev;
+	ctx->gmem = gdev;
+	ctx->mapping = inode->i_mapping;
+
+	lockdep_assert_held_write(&vdev->vdev.gmem_lock);
+	lockdep_assert_held(&vdev->vdev.dev_set->lock);
+	/*
+	 * Require access to be granted and the file to be bound to IOMMUFD.
+	 * The acquire pairs with smp_store_release() after vfio_df_open()
+	 * so the device setup is visible before proceeding.
+	 */
+	if (!smp_load_acquire(&df->access_granted) || !df->iommufd) {
+		ret = -EINVAL;
+		goto fail;
+	}
+
+	/*
+	 * Reject setup if dma-buf exports or ioeventfd handlers can access
+	 * BARs outside guest_memfd.
+	 */
+	if (!list_empty_careful(&vdev->dmabufs) || vdev->ioeventfds_nr) {
+		ret = -EBUSY;
+		goto fail;
+	}
+
+	/*
+	 * Reject setup while userspace allows the device to runtime suspend.
+	 */
+	if (READ_ONCE(vdev->pm_runtime_engaged)) {
+		ret = -EBUSY;
+		goto fail;
+	}
+
+	/*
+	 * Reject the request if any VFIO BAR mapping exists
+	 */
+	if (atomic_read(&vdev->mmap_count)) {
+		ret = -EBUSY;
+		goto fail;
+	}
+
+	/* Runtime suspend must stay excluded until provider release. */
+	ret = vfio_pci_core_pm_pin_d0(vdev);
+	if (ret)
+		goto fail;
+
+	ivdev = iommufd_device_attach_mmio_provider(vdev->vdev.iommufd_device,
+						    gdev->kvm, ctx,
+						    vfio_gmem_has_private_mmio,
+						    &gdev->binding.owner);
+	if (IS_ERR(ivdev)) {
+		ret = PTR_ERR(ivdev);
+		goto fail_pm;
+	}
+	gdev->binding.vdev_id = ivdev->virt_id;
+	ctx->ivdev = ivdev;
+	return ctx;
+fail_pm:
+	pm_runtime_put(&vdev->pdev->dev);
+fail:
+	kfree(ctx);
+	return ERR_PTR(ret);
+}
+
+/**
+ * vfio_gmem_prepare_memory_region() - validate a memslot BAR interval
+ * @data: attached provider context
+ * @offset: byte offset in the VFIO region namespace
+ * @size: range length in bytes
+ * @flags: requested KVM_MEM_LOG_DIRTY_PAGES and KVM_MEM_READONLY flags
+ *
+ * Check the entire interval and the BAR's existing iomap. BAR reservations
+ * are released by VFIO teardown.
+ *
+ * Return: 0 on success or a negative error code.
+ */
+static int vfio_gmem_prepare_memory_region(void *data,
+		u64 offset, u64 size, u32 flags)
+{
+	struct vfio_pci_gmem *ctx = data;
+	phys_addr_t pa;
+	u64 off;
+	u32 bar;
+	int ret;
+
+	if (flags & (KVM_MEM_LOG_DIRTY_PAGES | KVM_MEM_READONLY))
+		return -EINVAL;
+
+	ret = vfio_gmem_resolve_bar_range(ctx, offset, size, &bar, &off, &pa);
+	if (ret)
+		return ret;
+
+	return PTR_ERR_OR_ZERO(vfio_pci_core_get_iomap(ctx->vdev, bar));
+}
+
+/**
+ * vfio_gmem_get_pfn() - obtain a BAR PFN for guest_memfd
+ * @data: provider context returned by vfio_gmem_attach()
+ * @index: page index in the VFIO region namespace
+ * @pfn: receives the BAR PFN
+ * @max_order: receives zero for base-page mappings
+ *
+ * Called with the guest_memfd inode's invalidate_lock held. Take memory_lock
+ * for read while checking BAR access and resolving the PFN.
+ *
+ * Return: 0 on success, -ENXIO if BAR access is disabled, or a range or BAR
+ * mapping error.
+ */
+static int vfio_gmem_get_pfn(void *data, pgoff_t index,
+		unsigned long *pfn, int *max_order)
+{
+	struct vfio_pci_gmem *ctx = data;
+	struct vfio_pci_core_device *vdev = ctx->vdev;
+	phys_addr_t pa;
+	u64 off;
+	u32 bar;
+	int ret;
+
+	if ((u64)index >= (ctx->gmem->size >> PAGE_SHIFT))
+		return -EINVAL;
+
+	lockdep_assert_held(&ctx->mapping->invalidate_lock);
+	guard(rwsem_read)(&vdev->memory_lock);
+	if (!__vfio_pci_memory_enabled(vdev))
+		/* Retrying cannot change the decode state frozen at attachment. */
+		return -ENXIO;
+
+	ret = vfio_gmem_resolve_bar_range(ctx, (u64)index << PAGE_SHIFT, PAGE_SIZE,
+					  &bar, &off, &pa);
+	if (ret)
+		return ret;
+
+	/* Userspace can fault the BAR before a memslot has been installed. */
+	ret = PTR_ERR_OR_ZERO(vfio_pci_core_get_iomap(vdev, bar));
+	if (ret)
+		return ret;
+
+	*pfn = PHYS_PFN(pa);
+	if (max_order)
+		*max_order = 0;
+	return 0;
+}
+
+/**
+ * vfio_gmem_lock() - exclude TDI transitions during mapping changes
+ * @context: attached provider context
+ *
+ * Called with the inode invalidate lock held for write. Hold the vDEVICE
+ * MMIO mutex through guest invalidation, architecture conversion and
+ * publication of the private-range summary. Revocation, memslot unbind and
+ * final file cleanup use the same exclusion without changing attributes.
+ */
+static void vfio_gmem_lock(void *context)
+{
+	struct vfio_pci_gmem *ctx = context;
+
+	lockdep_assert_held_write(&ctx->mapping->invalidate_lock);
+	iommufd_vdevice_mmio_lock(ctx->ivdev);
+}
+
+/**
+ * vfio_gmem_unlock() - release provider exclusion after mapping changes
+ * @context: attached provider context
+ */
+static void vfio_gmem_unlock(void *context)
+{
+	struct vfio_pci_gmem *ctx = context;
+
+	lockdep_assert_held_write(&ctx->mapping->invalidate_lock);
+	lockdep_assert_held(&ctx->ivdev->mmio_lock);
+	iommufd_vdevice_mmio_unlock(ctx->ivdev);
+}
+
+/**
+ * vfio_gmem_release() - release a provider after successful core cleanup
+ * @data: provider context to release
+ *
+ * Detach from the pinned vDEVICE, drop runtime PM and free the context.
+ * Generic VFIO pins the source file through this call and clears attachment
+ * state after release returns.
+ *
+ * The caller holds gmem_lock for write and dev_set->lock. Detachment takes
+ * the vDEVICE MMIO lock.
+ */
+static void vfio_gmem_release(void *data)
+{
+	struct vfio_pci_gmem *ctx = data;
+	struct vfio_pci_core_device *vdev = ctx->vdev;
+
+	lockdep_assert_held_write(&vdev->vdev.gmem_lock);
+	lockdep_assert_held(&vdev->vdev.dev_set->lock);
+	iommufd_vdevice_detach_mmio_provider(ctx->ivdev);
+	pm_runtime_put(&vdev->pdev->dev);
+	kfree(ctx);
+}
+
+const struct guest_memfd_provider_operations vfio_pci_gmem_provider_ops = {
+	.attach = vfio_gmem_attach,
+	.prepare_memory_region = vfio_gmem_prepare_memory_region,
+	.device_memory = true,
+	.lock = vfio_gmem_lock,
+	.unlock = vfio_gmem_unlock,
+	.get_pfn = vfio_gmem_get_pfn,
+	.release = vfio_gmem_release,
+};
+EXPORT_SYMBOL_GPL(vfio_pci_gmem_provider_ops);
+
+MODULE_IMPORT_NS("IOMMUFD");
diff --git a/drivers/vfio/pci/vfio_pci_gmem.h b/drivers/vfio/pci/vfio_pci_gmem.h
new file mode 100644
index 000000000000..3ed53ae0c18a
--- /dev/null
+++ b/drivers/vfio/pci/vfio_pci_gmem.h
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef VFIO_PCI_GMEM_H
+#define VFIO_PCI_GMEM_H
+
+#include <linux/guest_memfd.h>
+
+struct iommufd_vdevice;
+struct vfio_pci_core_device;
+
+struct vfio_pci_gmem {
+	struct guest_memfd_device *gmem;
+	struct vfio_pci_core_device *vdev;
+	struct iommufd_vdevice *ivdev;
+	struct address_space *mapping;
+};
+
+#endif /* VFIO_PCI_GMEM_H */
diff --git a/drivers/vfio/pci/vfio_pci_priv.h b/drivers/vfio/pci/vfio_pci_priv.h
index 43d74c46e841..54e7c0fe0b0e 100644
--- a/drivers/vfio/pci/vfio_pci_priv.h
+++ b/drivers/vfio/pci/vfio_pci_priv.h
@@ -11,17 +11,26 @@
 /* Cap maximum number of ioeventfds per device (arbitrary) */
 #define VFIO_PCI_IOEVENTFD_MAX		1000
 
-/* Later patches reject BAR state changes while guest_memfd uses the device. */
+/*
+ * Exclusive BAR ownership excludes destructive state changes. File
+ * operations hold gmem_lock; reset of a device instead holds
+ * dev_set->lock. Either stabilizes gmem_attached, whose publication and
+ * removal hold both locks.
+ */
 static inline bool
 vfio_pci_bar_state_change_allowed(struct vfio_pci_core_device *vdev)
 {
-	return true;
+	lockdep_assert(lockdep_is_held(&vdev->vdev.gmem_lock) ||
+		       lockdep_is_held(&vdev->vdev.dev_set->lock));
+	return !vdev->vdev.gmem_attached;
 }
 
 static inline bool
 vfio_pci_bar_access_allowed(struct vfio_pci_core_device *vdev)
 {
-	return true;
+	lockdep_assert_held(&vdev->vdev.gmem_lock);
+
+	return !vdev->vdev.gmem_attached;
 }
 
 struct vfio_pci_ioeventfd {
diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index 4ebd29069034..485f953345bd 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -156,6 +156,10 @@ struct vfio_pci_core_device {
 	struct list_head	dmabufs;
 };
 
+#if IS_ENABLED(CONFIG_VFIO_PCI_GMEM)
+extern const struct guest_memfd_provider_operations vfio_pci_gmem_provider_ops;
+#endif
+
 enum vfio_pci_io_width {
 	VFIO_PCI_IO_WIDTH_1 = 1,
 	VFIO_PCI_IO_WIDTH_2 = 2,
-- 
2.43.0

[RFC PATCH v4 25/25] KVM: Document device-backed guest_memfd and VFIO usage

[Aneesh Kumar K.V (Arm)]

Describe device setup, VFIO access restrictions, mappings and lock ordering.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Alex Williamson <alex@shazbot.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: Kevin Tian <kevin.tian@intel.com>
Cc: "Joerg Roedel (AMD)" <joro@8bytes.org>
Cc: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 Documentation/virt/kvm/api.rst     | 50 ++++++++++++++++++++++++++----
 Documentation/virt/kvm/locking.rst | 27 ++++++++++++++++
 2 files changed, 71 insertions(+), 6 deletions(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index 4482d34ec281..289b42224258 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -6513,8 +6513,8 @@ The capability KVM_CAP_GUEST_MEMFD_FLAGS enumerates the `flags` that can be
 specified via KVM_CREATE_GUEST_MEMFD.  Currently defined flags:
 
   ============================= ================================================
-  GUEST_MEMFD_FLAG_MMAP         Enable using mmap() on the guest_memfd file
-                                descriptor.
+  GUEST_MEMFD_FLAG_MMAP         Enable userspace mappings through guest_memfd,
+                                including shared VFIO PCI BAR resources.
   GUEST_MEMFD_FLAG_INIT_SHARED  Make all memory in the file shared during
                                 KVM_CREATE_GUEST_MEMFD (memory files created
                                 without INIT_SHARED will be marked private).
@@ -6540,10 +6540,48 @@ provider. PFN_MODE selects the PFN callbacks; providers that supply device
 operations additionally require INIT_SHARED and a VM supporting protected
 device mappings.
 
-KVM_SET_MEMORY_ATTRIBUTES2 converts device memory using guest_memfd offsets.
-For private conversion, KVM resolves the guest address through the memslot
-binding and obtains the physical address from the provider. Shared conversion
-removes the protected device mappings before publishing shared attributes.
+For VFIO PCI device resources, resource_fd must refer to a VFIO device fd
+bound to IOMMUFD, with a vDEVICE belonging to the same VM. Set USE_RESOURCE,
+PFN_MODE and INIT_SHARED. File offsets use the VFIO region-offset namespace.
+Memslots must cover page-aligned ranges within a single memory-mapped I/O
+BAR that supports mmap. Dirty logging and read-only memslots are rejected
+with EINVAL.
+
+Creation fails with EBUSY if the device already supplies another guest_memfd,
+has VFIO BAR mappings, dma-buf exports or ioeventfd handlers, or userspace has
+enabled VFIO runtime suspend. The device is resumed and must be in D0; it is
+kept active until guest_memfd releases it.
+
+While guest_memfd uses the device, the following operations fail with EBUSY
+through any VFIO fd referring to it, even if all ranges are shared:
+
+- BAR read/write, mmap, new ioeventfd registration and dma-buf export.
+- VFIO_DEVICE_RESET, PCIe/AF function-level reset and hot reset affecting
+  the device.
+- Requests for D1, D2 or D3, enabling VFIO runtime suspend, and configuration
+  writes that change Memory Enable or Bus Master Enable.
+
+Rejected configuration writes leave the hardware and virtual registers
+unchanged. Unrelated configuration writes, queries and IRQ configuration
+remain available.
+
+GUEST_MEMFD_FLAG_MMAP permits MAP_SHARED mappings through guest_memfd.
+Without this flag, mmap fails with ENODEV.
+
+MSI-X SET_IRQS remains available. Userspace must keep the MSI-X table and
+PBA pages shared while host-managed MSI-X is in use and reject guest private
+mapping requests for those pages. The kernel does not enforce this policy.
+
+Use :ref:`KVM_SET_MEMORY_ATTRIBUTES2` to convert device memory by guest_memfd
+offset. Private conversion uses the memslot binding and provider PFNs to
+install protected device mappings before publishing private attributes.
+Shared conversion removes those mappings before publishing shared attributes.
+
+Revocation and memslot deletion remove shared and private mappings without
+changing file attributes. Private ranges still prevent device unlock after
+these operations; userspace must share them or release the last guest_memfd
+reference. Final release removes mappings and clears the private-range
+summary before releasing the device resources.
 
 When the KVM MMU performs a PFN lookup to service a guest fault, the fault will
 always be consumed from guest_memfd, regardless of whether it is a shared or a
diff --git a/Documentation/virt/kvm/locking.rst b/Documentation/virt/kvm/locking.rst
index 662231e958a0..a2a6b4dd9a1f 100644
--- a/Documentation/virt/kvm/locking.rst
+++ b/Documentation/virt/kvm/locking.rst
@@ -58,6 +58,33 @@ On x86:
   kvm->arch.tdp_mmu_pages_lock and kvm->arch.mmu_unsync_pages_lock must
   also take kvm->arch.mmu_lock
 
+For device-backed guest_memfd, conversion takes locks in this order:
+
+- inode->i_mapping->invalidate_lock for write
+- IOMMUFD vDEVICE mmio_lock
+- VFIO vdev->memory_lock for read, during PFN lookup
+
+The inode invalidate lock protects attributes and offset-to-GPA bindings.
+The provider's lock()/unlock() callbacks hold mmio_lock across
+invalidation, conversion and publication of attributes. Device unlock takes
+the same mutex when checking for private ranges. Range revocation, memslot
+unbind and final file cleanup use the same order.
+
+PFN lookup during faults takes the inode invalidate lock for read, then
+memory_lock for read, without taking mmio_lock. memory_lock protects the
+power and Memory Enable checks and BAR PFN lookup against PCI state
+restore. Host faults retain the inode invalidate lock through PTE insertion.
+
+VFIO provider setup and release take gmem_lock for write, then
+dev_set->lock, before taking mmio_lock. Both outer locks protect updates to
+gmem_attached. VFIO file operations read this state under gmem_lock for
+read; reset checks can instead use dev_set->lock.
+
+VFIO resumes the device and takes a runtime-PM reference before acquiring
+mmio_lock during provider setup. It drops that reference after removing
+the provider from the vDEVICE during release. Runtime PM callbacks do not
+take the guest_memfd inode invalidate lock.
+
 Everything else is a leaf: no other lock is taken inside the critical
 sections.
 
-- 
2.43.0