[PATCH v9 0/2] selftests/vfio: Add NVIDIA GPU Falcon DMA test driver

[Rubin Du <rubind@nvidia.com>]

This patch series adds a new VFIO selftest plugin driver for NVIDIA 
GPUs that enables DMA testing via the Falcon microcontrollers.

Patch 1: 

Kernel selftests are a collection of test programs that live within the
Linux kernel source tree (tools/testing/selftests/) and are designed to
test various kernel subsystems from userspace. The VFIO selftest 
framework have a pluggable driver architecture that allows different 
hardware drivers to implement various test capabilities. However, not 
all drivers can trigger MSI/MSI-X interrupts from software.

This patch adds checks to gracefully skip MSI-related tests when the
driver's send_msi callback is NULL, allowing drivers without MSI
support to still run the DMA functionality tests. It also makes MSI
truly optional by checking msi_fd validity before operations.

Patch 2:

This patch introduces the core implementation of the plugin driver. It 
extracts and adapts relevant functionality from NVIDIA's gpu-admin-tools 
project [1], integrating it into the VFIO selftest framework. As a 
result, any system equipped with a PCIe slot and a supported NVIDIA GPU 
can now run VFIO DMA selftests using commonly available hardware.

The Falcon is a general-purpose microcontroller present on NVIDIA GPUs
that can perform DMA operations between system memory and device memory.

The core VFIO selftest infrastructure handles:

- VFIO container/group management
- IOMMU domain setup
- DMA buffer allocation and mapping
- Test orchestration and reporting

The plugin drivers provide device-specific implementations for:

- Probing and initializing device
- Triggering DMA operations
- Verifying DMA completion
- Device cleanup

[1] https://github.com/NVIDIA/gpu-admin-tools

Changes in v9:
- Squashed patch 3 (PMU falcon support for Kepler and Maxwell Gen1)
  into patch 2, as the registers and fields required have been approved
  for open source disclosure

Changes in v8:
- Corrected Makefile to also build nv_falcons driver on other
  architectures than x86_64

Changes in v7:
- Added Hopper (H100) support
- Made MSI optional by checking msi_fd != -1 in ASSERT_NO_MSI
  macro and guarding fcntl_set_nonblock() calls
- Refactored to use gpu_properties_map[] array indexed by enum gpu_arch
- Added falcon_map[] array indexed by enum falcon_type for cleaner
  initialization
- Coding style fixes

Changes in v6:
- Added GPU architecture detection
- Refactored GPU detection to use per-architecture property structs

Changes in v5:
- Reorganized as a 3-patch series
- Added patch to skip MSI tests for drivers without MSI support
- Removed stub MSI function from Falcon driver
- Added support to Maxwell Gen1 GPUs and Kepler GPUs

Changes in v4:
- Removed redundant PCI_VENDOR_ID_NVIDIA macro
- Macro cleanup and style fixes

Changes in v3:
- Updated cover letter to clarify purpose and scope

Changes in v2:
- Fixed NV_PMC_ENABLE_PWR macro value (0x2000, was incorrectly 0x1000)
- Added gpu_disable_bus_master and falcon_disable calls in remove path
  for proper cleanup
- Added error handling for unknown GPU pmc_boot_0 values
- General code cleanup and style fixes
- Note: Kepler cards may not work, pending further testing

Rubin Du (2):
  selftests/vfio: Skip MSI tests for drivers that cannot raise
    interrupts
  selftests/vfio: Add NVIDIA Falcon driver for DMA testing

 .../vfio/lib/drivers/nv_falcons/hw.h          | 345 ++++++++
 .../vfio/lib/drivers/nv_falcons/nv_falcons.c  | 757 ++++++++++++++++++
 .../lib/include/libvfio/vfio_pci_device.h     |   3 +
 tools/testing/selftests/vfio/lib/libvfio.mk   |   2 +
 .../selftests/vfio/lib/vfio_pci_driver.c      |   4 +-
 .../selftests/vfio/vfio_pci_driver_test.c     |   8 +
 6 files changed, 1118 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/vfio/lib/drivers/nv_falcons/hw.h
 create mode 100644 tools/testing/selftests/vfio/lib/drivers/nv_falcons/nv_falcons.c

-- 
2.43.0