kvm.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [RFC PATCH 00/33] vfio: Introduce selftests for VFIO
@ 2025-05-23 23:29 David Matlack
  2025-05-23 23:29 ` [RFC PATCH 01/33] selftests: Create tools/testing/selftests/vfio David Matlack
                   ` (35 more replies)
  0 siblings, 36 replies; 64+ messages in thread
From: David Matlack @ 2025-05-23 23:29 UTC (permalink / raw)
  To: Alex Williamson
  Cc: Shuah Khan, Paolo Bonzini, David Matlack, Vinod Koul, Fenghua Yu,
	Masami Hiramatsu (Google), Adhemerval Zanella, Jiri Olsa,
	Andrii Nakryiko, Wei Yang, Bjorn Helgaas, Takashi Iwai,
	Greg Kroah-Hartman, Pierre-Louis Bossart, Andy Shevchenko,
	FUJITA Tomonori, WangYuli, Sean Christopherson, Andrew Jones,
	Claudio Imbrenda, Eric Auger, Josh Hilke, linux-kselftest, kvm,
	Jason Gunthorpe, Kevin Tian, Vipin Sharma, Pasha Tatashin,
	Saeed Mahameed, Adithya Jayachandran, Parav Pandit,
	Leon Romanovsky, Vinicius Costa Gomes, Dave Jiang, Dan Williams

This series introduces VFIO selftests, located in
tools/testing/selftests/vfio/.

VFIO selftests aim to enable kernel developers to write and run tests
that take the form of userspace programs that interact with VFIO and
IOMMUFD uAPIs. VFIO selftests can be used to write functional tests for
new features, regression tests for bugs, and performance tests for
optimizations.

These tests are designed to interact with real PCI devices, i.e. they do
not rely on mocking out or faking any behavior in the kernel. This
allows the tests to exercise not only VFIO but also IOMMUFD, the IOMMU
driver, interrupt remapping, IRQ handling, etc.

We chose selftests to host these tests primarily to enable integration
with the existing KVM selftests. As explained in the next section,
enabling KVM developers to test the interaction between VFIO and KVM is
one of the motivators of this series.

Motivation
-----------------------------------------------------------------------

The main motivation for this series is upcoming development in the
kernel to support Hypervisor Live Updates [1][2]. Live Update is a
specialized reboot process where selected devices are kept operational
and their kernel state is preserved and recreated across a kexec. For
devices, DMA and interrupts may continue during the reboot. VFIO-bound
devices are the main target, since the first usecase of Live Updates is
to enable host kernel upgrades in a Cloud Computing environment without
disrupting running customer VMs.

To prepare for upcoming support for Live Updates in VFIO, IOMMUFD, IOMMU
drivers, the PCI layer, etc., we'd like to first lay the ground work for
exercising and testing VFIO from kernel selftests. This way when we
eventually upstream support for Live Updates, we can also upstream tests
for those changes, rather than purely relying on Live Update integration
tests which would be hard to share and reproduce upstream.

But even without Live Updates, VFIO and IOMMUFD are becoming an
increasingly critical component of running KVM-based VMs in cloud
environments. Virtualized networking and storage are increasingly being
offloaded to smart NICs/cards, and demand for high performance
networking, storage, and AI are also leading to NICs, SSDs, and GPUs
being directly attached to VMs via VFIO.

VFIO selftests increases our ability to test in several ways.

 - It enables developers sending VFIO, IOMMUFD, etc. commits upstream to
   test their changes against all existing VFIO selftests, reducing the
   probability of regressions.

 - It enables developers sending VFIO, IOMMUFD, etc. commits upstream to
   include tests alongside their changes, increasing the quality of the
   code that is merged.

 - It enables testing the interaction between VFIO and KVM. There are
   some paths in KVM that are only exercised through VFIO, such as IRQ
   bypass. VFIO selftests provides a helper library to enable KVM
   developers to write KVM selftests to test those interactions [3].

Design
-----------------------------------------------------------------------

VFIO selftests are designed around interacting with with VFIO-managed PCI
devices. As such, the core data struture is struct vfio_pci_device, which
represents a single PCI device.

  struct vfio_pci_device *device;

  device = vfio_pci_device_init("0000:6a:01.0", iommu_mode);

  ...

  vfio_pci_device_cleanup(device);

vfio_pci_device_init() sets up a container or iommufd, depending on the
iommu_mode argument, to manage DMA mappings, fetches information about
the device and what interrupts it supports from VFIO and caches it, and
mmap()s all mappable BARs for the test to use.

There are helper methods that operate on struct vfio_pci_device to do
things like read and write to PCI config space, enable/disable IRQs, and
map memory for DMA,

struct vfio_pci_device and its methods do not care about what device
they are actually interacting with. It can be a GPU, a NIC, an SSD, etc.

To keep things simple initially, VFIO selftests only support a single
device per group and per container/iommufd. But it should be possible to
relax those restrictions in the future, e.g. to enable testing with
multiple devices in the same container/iommufd.

Driver Framework
-----------------------------------------------------------------------

In order to support VFIO selftests where a device is generating DMA and
interrupts on command, the VFIO selftests supports a driver framework.

This framework abstracts away device-specific details allowing VFIO
selftests to be written in a generic way, and then run against different
devices depending on what hardware developers have access to.

The framework also aims to support carrying drivers out-of-tree, e.g.
so that companies can run VFIO selftests with custom/test hardware.

Drivers must implement the following methods:

 - probe():        Check if the driver supports a given device.
 - init():         Initialize the driver.
 - remove():       Deinitialize the driver and reset the device.
 - memcpy_start(): Kick off a series of repeated memcpys (DMA reads and
                   DMA writes).
 - memcpy_wait():  Wait for a memcpy operation to complete.
 - send_msi():     Make the device send an MSI interrupt.

memcpy_start/wait() are for generating DMA. We separate the operation
into 2 steps so that tests can trigger a long-running DMA operation. We
expect to use this to stress test Live Updates by kicking off a
long-running mempcy operation and then performing a Live Update. These
methods are required to not generate any interrupts.

send_msi() is used for testing MSI and MSI-x interrupts. The driver
tells the test which MSI it will be using via device->driver.msi.

It's the responsibility of the test to set up a region of memory
and map it into the device for use by the driver, e.g. for in-memory
descriptors, before calling init().

A demo of the driver framework can be found in
tools/testing/selftests/vfio/vfio_pci_driver_test.c.

In addition, this series introduces a new KVM selftest to demonstrate
delivering a device MSI directly into a guest, which can be found in
tools/testing/selftests/kvm/vfio_pci_device_irq_test.c.

Tests
-----------------------------------------------------------------------

There are 5 tests in this series, mostly to demonstrate as a
proof-of-concept:

 - tools/testing/selftests/vfio/vfio_pci_device_test.c
 - tools/testing/selftests/vfio/vfio_pci_driver_test.c
 - tools/testing/selftests/vfio/vfio_iommufd_setup_test.c
 - tools/testing/selftests/vfio/vfio_dma_mapping_test.c
 - tools/testing/selftests/kvm/vfio_pci_device_irq_test.c

Integrating with KVM selftests
-----------------------------------------------------------------------

To support testing the interactions between VFIO and KVM, the VFIO
selftests support sharing its library with the KVM selftest. The patches
at the end of this series demonstrate how that works.

Essentially, we allow the KVM selftests to build their own copy of
tools/testing/selftests/vfio/lib/ and link it into KVM selftests
binaries. This requires minimal changes to the KVM selftests Makefile.

Future Areas of Development
-----------------------------------------------------------------------

Library:

 - Driver support for devices that can be used on AMD, ARM, and other
   platforms.
 - Driver support for a device available in QEMU VMs.
 - Support for tests that use multiple devices.
 - Support for IOMMU groups with multiple devices.
 - Support for multiple devices sharing the same container/iommufd.
 - Sharing TEST_ASSERT() macros and other common code between KVM
   and VFIO selftests.

Tests:

 - DMA mapping performance tests for BARs/HugeTLB/etc.
 - Live Update selftests.
 - Porting Sean's KVM selftest for posted interrupts to use the VFIO
   selftests library [3]

This series can also be found on GitHub:

  https://github.com/dmatlack/linux/tree/vfio/selftests/rfc

Cc: Alex Williamson <alex.williamson@redhat.com>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: Kevin Tian <kevin.tian@intel.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Vipin Sharma <vipinsh@google.com>
Cc: Josh Hilke <jrhilke@google.com>
Cc: Pasha Tatashin <pasha.tatashin@soleen.com>
Cc: Saeed Mahameed <saeedm@nvidia.com>
Cc: Saeed Mahameed <saeedm@nvidia.com>
Cc: Adithya Jayachandran <ajayachandra@nvidia.com>
Cc: Parav Pandit <parav@nvidia.com>
Cc: Leon Romanovsky <leonro@nvidia.com>
Cc: Vinicius Costa Gomes <vinicius.gomes@intel.com>
Cc: Dave Jiang <dave.jiang@intel.com>
Cc: Dan Williams <dan.j.williams@intel.com>

[1] https://lore.kernel.org/all/f35359d5-63e1-8390-619f-67961443bfe1@google.com/
[2] https://lore.kernel.org/all/20250515182322.117840-1-pasha.tatashin@soleen.com/
[3] https://lore.kernel.org/kvm/20250404193923.1413163-68-seanjc@google.com/

David Matlack (28):
  selftests: Create tools/testing/selftests/vfio
  vfio: selftests: Add a helper library for VFIO selftests
  vfio: selftests: Introduce vfio_pci_device_test
  tools headers: Add stub definition for __iomem
  tools headers: Import asm-generic MMIO helpers
  tools headers: Import x86 MMIO helper overrides
  tools headers: Import iosubmit_cmds512()
  tools headers: Import drivers/dma/ioat/{hw.h,registers.h}
  tools headers: Import drivers/dma/idxd/registers.h
  tools headers: Import linux/pci_ids.h
  vfio: selftests: Keep track of DMA regions mapped into the device
  vfio: selftests: Enable asserting MSI eventfds not firing
  vfio: selftests: Add a helper for matching vendor+device IDs
  vfio: selftests: Add driver framework
  vfio: sefltests: Add vfio_pci_driver_test
  vfio: selftests: Add driver for Intel CBDMA
  vfio: selftests: Add driver for Intel DSA
  vfio: selftests: Move helper to get cdev path to libvfio
  vfio: selftests: Encapsulate IOMMU mode
  vfio: selftests: Add [-i iommu_mode] option to all tests
  vfio: selftests: Add vfio_type1v2_mode
  vfio: selftests: Add iommufd_compat_type1{,v2} modes
  vfio: selftests: Add iommufd mode
  vfio: selftests: Make iommufd the default iommu_mode
  vfio: selftests: Add a script to help with running VFIO selftests
  KVM: selftests: Build and link sefltests/vfio/lib into KVM selftests
  KVM: selftests: Test sending a vfio-pci device IRQ to a VM
  KVM: selftests: Use real device MSIs in vfio_pci_device_irq_test

Josh Hilke (5):
  vfio: selftests: Test basic VFIO and IOMMUFD integration
  vfio: selftests: Move vfio dma mapping test to their own file
  vfio: selftests: Add test to reset vfio device.
  vfio: selftests: Use command line to set hugepage size for DMA mapping
    test
  vfio: selftests: Validate 2M/1G HugeTLB are mapped as 2M/1G in IOMMU

 MAINTAINERS                                   |    7 +
 tools/arch/x86/include/asm/io.h               |  101 +
 tools/arch/x86/include/asm/special_insns.h    |   27 +
 tools/include/asm-generic/io.h                |  482 +++
 tools/include/asm/io.h                        |   11 +
 tools/include/drivers/dma/idxd/registers.h    |  601 +++
 tools/include/drivers/dma/ioat/hw.h           |  270 ++
 tools/include/drivers/dma/ioat/registers.h    |  251 ++
 tools/include/linux/compiler.h                |    4 +
 tools/include/linux/io.h                      |    4 +-
 tools/include/linux/pci_ids.h                 | 3212 +++++++++++++++++
 tools/testing/selftests/Makefile              |    1 +
 tools/testing/selftests/kvm/Makefile.kvm      |    6 +-
 .../testing/selftests/kvm/include/kvm_util.h  |    4 +
 tools/testing/selftests/kvm/lib/kvm_util.c    |   21 +
 .../selftests/kvm/vfio_pci_device_irq_test.c  |  173 +
 tools/testing/selftests/vfio/.gitignore       |    7 +
 tools/testing/selftests/vfio/Makefile         |   20 +
 .../testing/selftests/vfio/lib/drivers/dsa.c  |  416 +++
 .../testing/selftests/vfio/lib/drivers/ioat.c |  235 ++
 .../selftests/vfio/lib/include/vfio_util.h    |  271 ++
 tools/testing/selftests/vfio/lib/libvfio.mk   |   26 +
 .../selftests/vfio/lib/vfio_pci_device.c      |  573 +++
 .../selftests/vfio/lib/vfio_pci_driver.c      |  126 +
 tools/testing/selftests/vfio/run.sh           |  110 +
 .../selftests/vfio/vfio_dma_mapping_test.c    |  239 ++
 .../selftests/vfio/vfio_iommufd_setup_test.c  |  133 +
 .../selftests/vfio/vfio_pci_device_test.c     |  195 +
 .../selftests/vfio/vfio_pci_driver_test.c     |  256 ++
 29 files changed, 7780 insertions(+), 2 deletions(-)
 create mode 100644 tools/arch/x86/include/asm/io.h
 create mode 100644 tools/arch/x86/include/asm/special_insns.h
 create mode 100644 tools/include/asm-generic/io.h
 create mode 100644 tools/include/asm/io.h
 create mode 100644 tools/include/drivers/dma/idxd/registers.h
 create mode 100644 tools/include/drivers/dma/ioat/hw.h
 create mode 100644 tools/include/drivers/dma/ioat/registers.h
 create mode 100644 tools/include/linux/pci_ids.h
 create mode 100644 tools/testing/selftests/kvm/vfio_pci_device_irq_test.c
 create mode 100644 tools/testing/selftests/vfio/.gitignore
 create mode 100644 tools/testing/selftests/vfio/Makefile
 create mode 100644 tools/testing/selftests/vfio/lib/drivers/dsa.c
 create mode 100644 tools/testing/selftests/vfio/lib/drivers/ioat.c
 create mode 100644 tools/testing/selftests/vfio/lib/include/vfio_util.h
 create mode 100644 tools/testing/selftests/vfio/lib/libvfio.mk
 create mode 100644 tools/testing/selftests/vfio/lib/vfio_pci_device.c
 create mode 100644 tools/testing/selftests/vfio/lib/vfio_pci_driver.c
 create mode 100755 tools/testing/selftests/vfio/run.sh
 create mode 100644 tools/testing/selftests/vfio/vfio_dma_mapping_test.c
 create mode 100644 tools/testing/selftests/vfio/vfio_iommufd_setup_test.c
 create mode 100644 tools/testing/selftests/vfio/vfio_pci_device_test.c
 create mode 100644 tools/testing/selftests/vfio/vfio_pci_driver_test.c


base-commit: a11a72229881d8ac1d52ea727101bc9c744189c1
prerequisite-patch-id: 3bae97c9e1093148763235f47a84fa040b512d04
-- 
2.49.0.1151.ga128411c76-goog


^ permalink raw reply	[flat|nested] 64+ messages in thread

end of thread, other threads:[~2025-07-07 17:43 UTC | newest]

Thread overview: 64+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-05-23 23:29 [RFC PATCH 00/33] vfio: Introduce selftests for VFIO David Matlack
2025-05-23 23:29 ` [RFC PATCH 01/33] selftests: Create tools/testing/selftests/vfio David Matlack
2025-05-23 23:29 ` [RFC PATCH 02/33] vfio: selftests: Add a helper library for VFIO selftests David Matlack
2025-06-03  7:52   ` liulongfang
2025-05-23 23:29 ` [RFC PATCH 03/33] vfio: selftests: Introduce vfio_pci_device_test David Matlack
2025-05-23 23:29 ` [RFC PATCH 04/33] vfio: selftests: Test basic VFIO and IOMMUFD integration David Matlack
2025-06-02 23:08   ` Alex Williamson
2025-06-03 15:53     ` David Matlack
2025-05-23 23:29 ` [RFC PATCH 05/33] vfio: selftests: Move vfio dma mapping test to their own file David Matlack
2025-05-23 23:29 ` [RFC PATCH 06/33] vfio: selftests: Add test to reset vfio device David Matlack
2025-05-23 23:29 ` [RFC PATCH 07/33] vfio: selftests: Use command line to set hugepage size for DMA mapping test David Matlack
2025-05-26 17:15   ` Jason Gunthorpe
2025-05-30 16:50     ` David Matlack
2025-05-30 17:25       ` Jason Gunthorpe
2025-06-06 23:05         ` David Matlack
2025-06-09  0:23           ` Jason Gunthorpe
2025-05-23 23:29 ` [RFC PATCH 08/33] vfio: selftests: Validate 2M/1G HugeTLB are mapped as 2M/1G in IOMMU David Matlack
2025-05-26 17:16   ` Jason Gunthorpe
2025-05-27 23:04     ` David Matlack
2025-05-30 21:12   ` David Matlack
2025-05-31 19:09     ` Andy Shevchenko
2025-06-01  7:45       ` Greg Kroah-Hartman
2025-06-03 17:28         ` David Matlack
2025-05-23 23:29 ` [RFC PATCH 09/33] tools headers: Add stub definition for __iomem David Matlack
2025-05-23 23:29 ` [RFC PATCH 10/33] tools headers: Import asm-generic MMIO helpers David Matlack
2025-05-23 23:29 ` [RFC PATCH 11/33] tools headers: Import x86 MMIO helper overrides David Matlack
2025-05-23 23:29 ` [RFC PATCH 12/33] tools headers: Import iosubmit_cmds512() David Matlack
2025-05-23 23:29 ` [RFC PATCH 13/33] tools headers: Import drivers/dma/ioat/{hw.h,registers.h} David Matlack
2025-05-26 17:18   ` Jason Gunthorpe
2025-05-27 23:10     ` David Matlack
2025-05-23 23:29 ` [RFC PATCH 14/33] tools headers: Import drivers/dma/idxd/registers.h David Matlack
2025-05-23 23:30 ` [RFC PATCH 15/33] tools headers: Import linux/pci_ids.h David Matlack
2025-05-23 23:30 ` [RFC PATCH 16/33] vfio: selftests: Keep track of DMA regions mapped into the device David Matlack
2025-05-23 23:30 ` [RFC PATCH 17/33] vfio: selftests: Enable asserting MSI eventfds not firing David Matlack
2025-06-30 12:24   ` [PATCH RFC " Sairaj Kodilkar
2025-07-07 17:43     ` David Matlack
2025-05-23 23:30 ` [RFC PATCH 18/33] vfio: selftests: Add a helper for matching vendor+device IDs David Matlack
2025-05-23 23:30 ` [RFC PATCH 19/33] vfio: selftests: Add driver framework David Matlack
2025-05-23 23:30 ` [RFC PATCH 20/33] vfio: sefltests: Add vfio_pci_driver_test David Matlack
2025-05-23 23:30 ` [RFC PATCH 21/33] vfio: selftests: Add driver for Intel CBDMA David Matlack
2025-05-23 23:30 ` [RFC PATCH 22/33] vfio: selftests: Add driver for Intel DSA David Matlack
2025-05-23 23:30 ` [RFC PATCH 23/33] vfio: selftests: Move helper to get cdev path to libvfio David Matlack
2025-05-23 23:30 ` [RFC PATCH 24/33] vfio: selftests: Encapsulate IOMMU mode David Matlack
2025-05-23 23:30 ` [RFC PATCH 25/33] vfio: selftests: Add [-i iommu_mode] option to all tests David Matlack
2025-05-23 23:30 ` [RFC PATCH 26/33] vfio: selftests: Add vfio_type1v2_mode David Matlack
2025-05-23 23:30 ` [RFC PATCH 27/33] vfio: selftests: Add iommufd_compat_type1{,v2} modes David Matlack
2025-05-23 23:30 ` [RFC PATCH 28/33] vfio: selftests: Add iommufd mode David Matlack
2025-05-23 23:30 ` [RFC PATCH 29/33] vfio: selftests: Make iommufd the default iommu_mode David Matlack
2025-05-26 17:20   ` Jason Gunthorpe
2025-06-03 17:24     ` David Matlack
2025-05-23 23:30 ` [RFC PATCH 30/33] vfio: selftests: Add a script to help with running VFIO selftests David Matlack
2025-06-02 23:07   ` Alex Williamson
2025-06-03 15:51     ` David Matlack
2025-05-23 23:30 ` [RFC PATCH 31/33] KVM: selftests: Build and link sefltests/vfio/lib into KVM selftests David Matlack
2025-05-23 23:30 ` [RFC PATCH 32/33] KVM: selftests: Test sending a vfio-pci device IRQ to a VM David Matlack
2025-05-23 23:30 ` [RFC PATCH 33/33] KVM: selftests: Use real device MSIs in vfio_pci_device_irq_test David Matlack
2025-05-26  9:12 ` [RFC PATCH 00/33] vfio: Introduce selftests for VFIO Yi Liu
2025-05-27 17:26   ` David Matlack
2025-05-26 17:09 ` Jason Gunthorpe
2025-05-27 23:01   ` David Matlack
2025-05-28  0:11     ` Jason Gunthorpe
2025-05-28 21:52       ` David Matlack
2025-06-03  7:11     ` Joel Granados
2025-06-02 17:12 ` Cédric Le Goater

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).