[RFC PATCH 0/7] vfio: selftests: Add peer-to-peer DMA selftest

[David Matlack <dmatlack@google.com>]

This series can be found on GitHub:

  https://github.com/dmatlack/linux/tree/vfio/selftests/p2pdma/rfc/v1

This series adds a peer-to-peer (P2P) DMA selftest to the VFIO
selftests. The test pairs an initiator device supported by the VFIO
selftests driver framework (dsa, igb, ioat, mlx5, or nv_falcon) with a
target device that exposes a memory BAR, maps the target's BAR into the
IOMMU, and uses the initiator to perform DMA reads and writes targeting
the BAR.

No changes to struct vfio_pci_driver_ops are needed. Because
vfio_pci_driver_memcpy() already takes source and destination IOVAs, any
existing selftests driver can drive P2P DMA once a peer's BAR is mapped
into its IOMMU address space.

There are two motivations for adding P2P DMA support to VFIO selftests:

  1. Exercise the P2P DMA paths in VFIO and iommufd end-to-end,
     including mapping BAR mmaps via VFIO_IOMMU_MAP_DMA in legacy
     type1/type1v2 modes and exporting BARs via
     VFIO_DEVICE_FEATURE_DMA_BUF + IOMMU_IOAS_MAP_FILE (and dma-buf
     revocation) in native iommufd.

  2. Lay the groundwork for testing uninterrupted P2P DMA across a Live
     Update in the future, building on VFIO PCI Live Update support [1]
     and the PCI core changes to keep preserved devices accessible
     across kexec [2].

Patch 1 comes from v3 of Mina's huge_fault series [3] and is included
here for convenience. It adds vfio_pci_find_mem_bar() to locate a BAR
range that behaves like ordinary memory, starting with QEMU's
pci-testdev.

This series applies on top of vfio/next (vfio-v7.4-rc1).

[1] https://lore.kernel.org/kvm/20261102153656.1431701-1-vipinsh@google.com/
[2] https://lore.kernel.org/linux-pci/20261019213054.3157159-1-dmatlack@google.com/
[3] https://lore.kernel.org/kvm/20261102230445.3993629-3-almasrymina@google.com/

Series Overview
---------------

This series implements the following:

  1. Add vfio_pci_find_mem_bar() to locate a BAR range that can be
     safely read and written like memory, starting with QEMU's
     pci-testdev, and verify that writes stick before returning it so
     that unbacked pci-testdev BARs are skipped (patches 1-2).

  2. Allow VFIO_SELFTESTS_BDF to contain multiple whitespace-separated
     BDFs so multi-device tests can run via run_kselftest.sh while
     single-device tests continue to use the first BDF (patch 3).

  3. Add vfio_pci_dmabuf_create() and iommu_map_file() helpers to export
     a VFIO PCI region range via VFIO_DEVICE_FEATURE_DMA_BUF and map it
     into an iommufd IOAS via IOMMU_IOAS_MAP_FILE (patches 4-5).

  4. Add vfio_pci_find_p2p_pair() to find a compatible (initiator,
     target) pair from an array of VFIO PCI devices, providing a central
     place to check P2P topology reachability in the future (patch 6).

  5. Add vfio_pci_p2pdma_test, covering RAM-to-BAR, BAR-to-RAM,
     BAR-to-BAR, and RAM-BAR-RAM roundtrip copies across all supported
     IOMMU modes, plus iommufd dma-buf revocation when
     PCI_COMMAND_MEMORY is cleared on the target (patch 7).

Open Questions
--------------

  - Re-mapping after dma-buf revocation: Clearing PCI_COMMAND_MEMORY on
    the target device calls vfio_pci_dma_buf_move(vdev, true), which
    notifies iommufd via iopt_revoke_notify() and unmaps the BAR from
    the IOMMU domain. Re-enabling PCI_COMMAND_MEMORY calls
    vfio_pci_dma_buf_move(vdev, false), allowing the dma-buf to be
    mapped again, but dma-buf has no callback to re-populate iommufd, so
    the existing IOAS mapping stays unmapped until userspace calls
    IOMMU_IOAS_UNMAP and IOMMU_IOAS_MAP_FILE again.

    Is requiring userspace to re-issue IOMMU_IOAS_MAP_FILE the expected
    uAPI behavior? VFIO_DEVICE_RESET (and runtime PM transitions) on the
    target device also bracket the reset with
    vfio_pci_dma_buf_move(vdev, true) and vfio_pci_dma_buf_move(vdev,
    false), tearing down any active IOMMU_IOAS_MAP_FILE mapping of that
    device's BAR even though PCI_COMMAND_MEMORY remains enabled when
    VFIO_DEVICE_RESET returns.

  - Physical target devices and P2P topology checks: mem_bar_descs[]
    currently only matches QEMU's pci-testdev, and
    vfio_pci_has_p2p_path() does not yet check PCI topology or ACS to
    verify that P2P traffic can flow between two physical endpoints.

    Suggestions on physical target BARs to add to mem_bar_descs[] and on
    checking P2P reachability from userspace are welcome.

Testing
-------

All patches were build-tested individually. The series was tested on
x86_64 in QEMU 11.1 (membar-backed=on requires QEMU 10.1+) using an
emulated Intel 82576 NIC (igb) and a pci-testdev with a 1G RAM-backed
BAR2 behind an emulated Intel IOMMU:

  qemu-system-x86_64 \
    -M q35,accel=kvm,kernel-irqchip=split \
    -device intel-iommu,intremap=on,caching-mode=on,device-iotlb=on \
    -netdev user,id=net0 \
    -device pcie-root-port,id=rp0,slot=2 \
    -device igb,bus=rp0,addr=0.0,netdev=net0 \
    -device pci-testdev,membar=1G,membar-backed=on \
    ...

Inside the guest, bind both devices to vfio-pci and run the test (BDF
order does not matter):

  # tools/testing/selftests/vfio/scripts/setup.sh 0000:01:00.0 0000:00:0d.0
  # tools/testing/selftests/vfio/scripts/run.sh \
      tools/testing/selftests/vfio/vfio_pci_p2pdma_test

Or via run_kselftest.sh:

  # VFIO_SELFTESTS_BDF="0000:01:00.0 0000:00:0d.0" \
      ./run_kselftest.sh -t vfio:vfio_pci_p2pdma_test


David Matlack (6):
  vfio: selftests: Verify memory behavior in vfio_pci_find_mem_bar()
  vfio: selftests: Allow multiple BDFs in VFIO_SELFTESTS_BDF
  vfio: selftests: Add vfio_pci_dmabuf_create() helper
  vfio: selftests: Add iommu_map_file() helper
  vfio: selftests: Add vfio_pci_find_p2p_pair() helper
  vfio: selftests: Add P2P DMA selftest

Mina Almasry (1):
  vfio: selftests: Add vfio_pci_find_mem_bar() helper

 tools/testing/selftests/vfio/Makefile         |   1 +
 .../vfio/lib/include/libvfio/iommu.h          |   9 +
 .../lib/include/libvfio/vfio_pci_device.h     |  23 +
 tools/testing/selftests/vfio/lib/iommu.c      |  29 ++
 tools/testing/selftests/vfio/lib/libvfio.c    |  53 ++-
 .../selftests/vfio/lib/vfio_pci_device.c      | 181 ++++++++
 .../selftests/vfio/vfio_pci_p2pdma_test.c     | 435 ++++++++++++++++++
 7 files changed, 717 insertions(+), 14 deletions(-)
 create mode 100644 tools/testing/selftests/vfio/vfio_pci_p2pdma_test.c


base-commit: 076d7e989cb69f6ded7d60f2a080eb909930061b
-- 
2.56.0.567.gdb80ae8cd5-goog