Re: [RFC PATCH v4 06/25] KVM: guest_memfd: Support device-backed PFN providers

[Fred Griffoul <griffoul@gmail.com>]

"Aneesh Kumar K.V (Arm)" <aneesh.kumar@kernel.org> writes:

> Device resources such as PCI BARs supply PFNs directly and cannot use
> the folio allocation path. Store the mapping owner and device ID in the
> core context during attach(), and use them when preparing device
> memslots. Keep device memory separate from ordinary PFN-backed RAM.

The device rules you wrote down in api.rst and locking.rst helped a
lot. If I read v4 right, the device backing differs from the RAM PFN
backing in these places:

  1. PFN mode calls RAM arch hooks: kvm_arch_gmem_make_shared() at
     conversion, kvm_arch_gmem_make_private() at fault, and
     kvm_arch_gmem_reclaim() on revoke and release. The device path
     has to skip them.
  2. lookup_pfn_range() only feeds those hooks, so the device path
     does not need it.
  3. The host mmap and stage-2 need a device memory type.
  4. In PFN mode, a conversion to private only updates the attributes,
     and the guest fault maps the page later. CCA has to map device
     memory at conversion, because the guest validates the mapping
     before it touches it.
  5. The device provider needs lock()/unlock(), the private range
     summary and prepare_memory_region().
  6. The provider gets the VM and the vDEVICE binding at attach.

Would these changes to PFN mode be enough to drop the device table?

  1. The arch hooks take the frame attributes. For RAM the arch does
     the RMP work. For MMIO it skips it, or does the RMM work instead.
  2. lookup_pfn_range() stays required. For vfio-pci it is the BAR
     start plus the offset.
  3. get_pfn() and lookup_pfn_range() report a memory type (MMIO) in
     their attributes, next to read only. The core mmap and the KVM
     fault path use it.
  4. A conversion calls an arch hook in both directions, and the hook
     can fail. CCA maps the private device range there.
  5. lock()/unlock(), the private range summary and
     prepare_memory_region() become optional ops for any PFN provider.
     A provider that moves RAM between guests needs them too: it has
     to order its moves with conversions and leave private pages
     alone.

I'm less sure about 6. Could KVM ask the provider for the vDEVICE when
the memslot is created, passing the VM? The provider would then not
need to keep a VM pointer.

Is there something in the device path that this would not cover?

Thanks,
Fred