RFC iommutests_: Testing software for everything IOMMU

RFC iommutests_: Testing software for everything IOMMU

[Joel Granados]

Hello everyone

This is a Request For Comment (RFC) describing a new testing infra
called iommutests. It is motivated by interest expressed during the "IO
Page Fault for all" [1] talk in LPC 2024 [2]. At the end of that
presentation in the "VFIO/IOMMU/PCI MC" microconference, There was a
clear "yes please" after asking if it made sense to have an
infrastructure that could evaluate the full spectrum of IOMMU
interactions — from host software to IOMMU in hardware and in emulated
environments.

What is iommutests?
--------------------
The primary goal of iommutests [3] is to provide a modular framework to
test IOMMU-mediated Direct Memory Access (DMA). It is meant to run
PASS/NO_PASS tests for the interaction between kernel subsystems and
hardware/virtual devices. All this in the hope of clarifying which IOMMU
features are working correctly and which are not. You can find the
project in github : https://github.com/SamsungDS/iommutests

* Test Orchestration:
  This component is implemented in pytest [4], a testing framework where
  tests can be parametrized, filtered and implemented concisely and
  easily. Additionally, pyudev [5] is employed for device enumeration
  and monitoring, as well as querying device properties and attributes.

* Test Executables:
  For the creation and execution of tests, iommutests leans heavily on
  libvfn [6], a zero-dependency C library designed for interacting with
  PCIe-based devices from user-space using the Linux kernel user API.
  The libvfn library can be used to abstract away common lower-level
  interactions which can then be re-used through out.

A Working Demonstration
-----------------------
To better illustrate how everything fits together, There’s a demo script
[7], with some demo notes [8] explaining the setup and output. It does
the following:

1. Sets up a custom qemu [9] virtual machine environment using a custom
   test device.
2. Builds both libvfn and iommutests.
3. Runs a pair of example tests (one of them always fails to show how
   how that would look like)

This demo shows how everything comes together to give the pass/no-pass
IOMMU testing results. It is there for anyone interested in trying it
out or contributing. To run it, you need to pass it a local ssh key and
linux kernel compiled with VFIO support

Custom qemu device: pci-ats-testdev
-------------------------------------
To support IOMMU testing under qemu, the pci-ats-testdev [10]
(different from pci-testdev [11]) was used to emulate DMA transactions.
It is a full fledged pci device capable of executing emulated DMA
accesses. It was originally intended to test Linux kernel interactions
with devices that had a working Address Translation Cache (ATC) but can
become a platform capable of testing anything PCI/IOMMU related if
needed.

Feedback
--------
This is a first draft, and many implementation details are still open to
refinement. I would appreciate your thoughts on any part of the project
— its design, scope, implementation language choices, or usability.
These are however some of the questions that are still outstanding from
my POV:

Q1: Beyond binary testing:
    Would iommutests be used for something other than pass/no-pass
    tests? Like performance? Stress testing?

Q2: Kernel Integration:
    Should iommutests interact with the already existing IOMMU linux
    kernel unit tests? Since it is an orchestration framework, then it
    can execute the unit tests in some way. This could actually be the
    next step, if found useful.

Thanks for your time
Best regards,

-- 

Joel Granados

[1] https://www.youtube.com/watch?v=UFrAjJ5TUf4
[2] https://lpc.events/event/18/timetable/#20240918
[3] https://github.com/SamsungDS/iommutests
[4] https://docs.pytest.org/en/stable/
[5] https://github.com/pyudev/pyudev
[6] https://github.com/SamsungDS/libvfn
[7] https://github.com/SamsungDS/iommutests/blob/master/docs/demo0_debian.sh
[8] https://github.com/SamsungDS/iommutests/blob/master/docs/demo0.md
[9] https://github.com/Joelgranados/qemu/tree/pcie-testdev
[10] https://github.com/Joelgranados/qemu/blob/pcie-testdev/hw/misc/pcie-ats-testdev.c
[11] https://github.com/Joelgranados/qemu/blob/pcie-testdev/hw/misc/pci-testdev.c

PS:
  Apologies for the long CC list. I always try to include the ppl that I
  think will be interested, but its not always easy to know who you are
  :). In this case, you are included because you appear as "M" under a
  subsystem that contains the string "IOMMU" in the MAINTAINERS file. Feel
  free to shoot me a mail if you don't want to be included in the future,
  I'll try my best to remove you.

Re: RFC iommutests_: Testing software for everything IOMMU

[Jean-Philippe Brucker]

On Fri, Mar 28, 2025 at 10:11:13AM +0100, Joel Granados wrote:
> Custom qemu device: pci-ats-testdev
> -------------------------------------
> To support IOMMU testing under qemu, the pci-ats-testdev [10]
> (different from pci-testdev [11]) was used to emulate DMA transactions.
> It is a full fledged pci device capable of executing emulated DMA
> accesses. It was originally intended to test Linux kernel interactions
> with devices that had a working Address Translation Cache (ATC) but can
> become a platform capable of testing anything PCI/IOMMU related if
> needed.

Yes please!  Maybe "pcie-testdev" rather than "pci-ats-testdev"?  There
are other PCIe features that are poorly tested at the moment, for example
PASID and PRI. The programming model of devices that actually implement
those can get too complex so we need something simpler to precisely stress
the IOMMU driver infrastructure. Driver unit-tests alone aren't good
enough for exercising TLB invalidation (DMA after removing a mapping must
crash), tricky cleanup paths (eg. killing a process bound to a device
that's issuing page requests), runtime PM, MSIs etc. I'm guessing testing
newer/future features like TDISP would also benefit from a simple device.

Some time back I needed a device like that to reproduce some tricky races
but never got round to implementing extra PCIe features. Although this one
[1] is based on virtio any programming interface should work as long as it
can instruct the device to send precise DMA transactions, ideally many in
parallel.

Thanks,
Jean

[1] https://jpbrucker.net/git/linux/log/?h=virtio-dmatest/latest
    https://jpbrucker.net/git/qemu/log/?h=virtio-dmatest/latest

Re: RFC iommutests_: Testing software for everything IOMMU

[Joel Granados]

On Tue, Apr 01, 2025 at 11:35:19AM +0100, Jean-Philippe Brucker wrote:
> On Fri, Mar 28, 2025 at 10:11:13AM +0100, Joel Granados wrote:
> > Custom qemu device: pci-ats-testdev
> > -------------------------------------
> > To support IOMMU testing under qemu, the pci-ats-testdev [10]
> > (different from pci-testdev [11]) was used to emulate DMA transactions.
> > It is a full fledged pci device capable of executing emulated DMA
> > accesses. It was originally intended to test Linux kernel interactions
> > with devices that had a working Address Translation Cache (ATC) but can
> > become a platform capable of testing anything PCI/IOMMU related if
> > needed.
> 
> Yes please!  Maybe "pcie-testdev" rather than "pci-ats-testdev"?  There
Definitely. If it is a more general pcie test framework, we need to
change the name to something like that; agreed.

> are other PCIe features that are poorly tested at the moment, for example
> PASID and PRI. The programming model of devices that actually implement
Actually, PRI was what we used it for. I have this as one of the
potential next steps for iommutests.

> those can get too complex so we need something simpler to precisely stress
> the IOMMU driver infrastructure. Driver unit-tests alone aren't good
> enough for exercising TLB invalidation (DMA after removing a mapping must
> crash), tricky cleanup paths (eg. killing a process bound to a device
> that's issuing page requests), runtime PM, MSIs etc. I'm guessing testing
Totally agree. PRI is tricky to test indeed.

> newer/future features like TDISP would also benefit from a simple device.
> 
> Some time back I needed a device like that to reproduce some tricky races
> but never got round to implementing extra PCIe features. Although this one
> [1] is based on virtio any programming interface should work as long as it
> can instruct the device to send precise DMA transactions, ideally many in
> parallel.
And it can be up-streamed to QEMU if it ends up being used for linux
kernel testing.

Best

-- 

Joel Granados