[RFC PATCH v4 25/25] KVM: Document device-backed guest_memfd and VFIO usage

["Aneesh Kumar K.V (Arm)" <aneesh.kumar@kernel.org>]

Describe device setup, VFIO access restrictions, mappings and lock ordering.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Alex Williamson <alex@shazbot.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: Kevin Tian <kevin.tian@intel.com>
Cc: "Joerg Roedel (AMD)" <joro@8bytes.org>
Cc: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 Documentation/virt/kvm/api.rst     | 50 ++++++++++++++++++++++++++----
 Documentation/virt/kvm/locking.rst | 27 ++++++++++++++++
 2 files changed, 71 insertions(+), 6 deletions(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index 4482d34ec281..289b42224258 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -6513,8 +6513,8 @@ The capability KVM_CAP_GUEST_MEMFD_FLAGS enumerates the `flags` that can be
 specified via KVM_CREATE_GUEST_MEMFD.  Currently defined flags:
 
   ============================= ================================================
-  GUEST_MEMFD_FLAG_MMAP         Enable using mmap() on the guest_memfd file
-                                descriptor.
+  GUEST_MEMFD_FLAG_MMAP         Enable userspace mappings through guest_memfd,
+                                including shared VFIO PCI BAR resources.
   GUEST_MEMFD_FLAG_INIT_SHARED  Make all memory in the file shared during
                                 KVM_CREATE_GUEST_MEMFD (memory files created
                                 without INIT_SHARED will be marked private).
@@ -6540,10 +6540,48 @@ provider. PFN_MODE selects the PFN callbacks; providers that supply device
 operations additionally require INIT_SHARED and a VM supporting protected
 device mappings.
 
-KVM_SET_MEMORY_ATTRIBUTES2 converts device memory using guest_memfd offsets.
-For private conversion, KVM resolves the guest address through the memslot
-binding and obtains the physical address from the provider. Shared conversion
-removes the protected device mappings before publishing shared attributes.
+For VFIO PCI device resources, resource_fd must refer to a VFIO device fd
+bound to IOMMUFD, with a vDEVICE belonging to the same VM. Set USE_RESOURCE,
+PFN_MODE and INIT_SHARED. File offsets use the VFIO region-offset namespace.
+Memslots must cover page-aligned ranges within a single memory-mapped I/O
+BAR that supports mmap. Dirty logging and read-only memslots are rejected
+with EINVAL.
+
+Creation fails with EBUSY if the device already supplies another guest_memfd,
+has VFIO BAR mappings, dma-buf exports or ioeventfd handlers, or userspace has
+enabled VFIO runtime suspend. The device is resumed and must be in D0; it is
+kept active until guest_memfd releases it.
+
+While guest_memfd uses the device, the following operations fail with EBUSY
+through any VFIO fd referring to it, even if all ranges are shared:
+
+- BAR read/write, mmap, new ioeventfd registration and dma-buf export.
+- VFIO_DEVICE_RESET, PCIe/AF function-level reset and hot reset affecting
+  the device.
+- Requests for D1, D2 or D3, enabling VFIO runtime suspend, and configuration
+  writes that change Memory Enable or Bus Master Enable.
+
+Rejected configuration writes leave the hardware and virtual registers
+unchanged. Unrelated configuration writes, queries and IRQ configuration
+remain available.
+
+GUEST_MEMFD_FLAG_MMAP permits MAP_SHARED mappings through guest_memfd.
+Without this flag, mmap fails with ENODEV.
+
+MSI-X SET_IRQS remains available. Userspace must keep the MSI-X table and
+PBA pages shared while host-managed MSI-X is in use and reject guest private
+mapping requests for those pages. The kernel does not enforce this policy.
+
+Use :ref:`KVM_SET_MEMORY_ATTRIBUTES2` to convert device memory by guest_memfd
+offset. Private conversion uses the memslot binding and provider PFNs to
+install protected device mappings before publishing private attributes.
+Shared conversion removes those mappings before publishing shared attributes.
+
+Revocation and memslot deletion remove shared and private mappings without
+changing file attributes. Private ranges still prevent device unlock after
+these operations; userspace must share them or release the last guest_memfd
+reference. Final release removes mappings and clears the private-range
+summary before releasing the device resources.
 
 When the KVM MMU performs a PFN lookup to service a guest fault, the fault will
 always be consumed from guest_memfd, regardless of whether it is a shared or a
diff --git a/Documentation/virt/kvm/locking.rst b/Documentation/virt/kvm/locking.rst
index 662231e958a0..a2a6b4dd9a1f 100644
--- a/Documentation/virt/kvm/locking.rst
+++ b/Documentation/virt/kvm/locking.rst
@@ -58,6 +58,33 @@ On x86:
   kvm->arch.tdp_mmu_pages_lock and kvm->arch.mmu_unsync_pages_lock must
   also take kvm->arch.mmu_lock
 
+For device-backed guest_memfd, conversion takes locks in this order:
+
+- inode->i_mapping->invalidate_lock for write
+- IOMMUFD vDEVICE mmio_lock
+- VFIO vdev->memory_lock for read, during PFN lookup
+
+The inode invalidate lock protects attributes and offset-to-GPA bindings.
+The provider's lock()/unlock() callbacks hold mmio_lock across
+invalidation, conversion and publication of attributes. Device unlock takes
+the same mutex when checking for private ranges. Range revocation, memslot
+unbind and final file cleanup use the same order.
+
+PFN lookup during faults takes the inode invalidate lock for read, then
+memory_lock for read, without taking mmio_lock. memory_lock protects the
+power and Memory Enable checks and BAR PFN lookup against PCI state
+restore. Host faults retain the inode invalidate lock through PTE insertion.
+
+VFIO provider setup and release take gmem_lock for write, then
+dev_set->lock, before taking mmio_lock. Both outer locks protect updates to
+gmem_attached. VFIO file operations read this state under gmem_lock for
+read; reset checks can instead use dev_set->lock.
+
+VFIO resumes the device and takes a runtime-PM reference before acquiring
+mmio_lock during provider setup. It drops that reference after removing
+the provider from the vDEVICE during release. Runtime PM callbacks do not
+take the guest_memfd inode invalidate lock.
+
 Everything else is a leaf: no other lock is taken inside the critical
 sections.
 
-- 
2.43.0