[RFC PATCH v2 vfio 0/7] pds vfio driver

Brett Creeley
2022-12-14T23:23:09Z

This is a first draft patchset for a new vendor specific VFIO driver for
use with the AMD/Pensando Distributed Services Card (DSC). This driver
(pds_vfio) is a client of the newly introduced pds_core driver.

Reference to the pds_core patchset:
https://lore.kernel.org/netdev/20221207004443.33779-1-shannon.nelson@amd.com/

AMD/Pensando already supports a NVMe VF device (1dd8:1006) in the
Distributed Services Card (DSC). This patchset adds the new pds_vfio
driver in order to support NVMe VF live migration.

This driver will use the pds_core device and auxiliary_bus as the VFIO
control path to the DSC. The pds_core device creates auxiliary_bus devices
for each live migratable VF. The devices are named by their feature plus
the VF PCI BDF so the auxiliary_bus driver implemented by pds_vfio can find
its related VF PCI driver instance. Once this auxiliary bus connection
is configured, the pds_vfio driver can send admin queue commands to the
device and receive events from pds_core.

An ASCII diagram of a VFIO instance looks something like this and can
be used with the VFIO subsystem to provide devices VFIO and live
migration support.

.------. .--------------------------.
| QEMU |--| VM .-------------. |
'......' | | nvme driver | |
| | .-------------. |
| | | SR-IOV VF | |
| | '-------------' |
| '---------------||---------'
.--------------. ||
|/dev/<vfio_fd>| ||
'--------------' ||
Host Userspace | ||
=================================================== ||
Host Kernel | ||
| ||
pds_core.LM.2305 <--+ .--------. ||
| | |vfio-pci| ||
| | '--------' ||
| | | ||
.------------. .-------------. ||
| pds_core | | pds_vfio | ||
'------------' '-------------' ||
|| || ||
09:00.0 09:00.1 ||
== PCI ==================================================||=====
|| || ||
.----------. .----------. ||
,-----| PF |---------| VF |-------------------,
| '----------' '----------' | nvme |
| DSC | data/control |
| | path |
-----------------------------------------------------------

The pds_vfio driver is targeted to reside in drivers/vfio/pci/pds.
It makes use of and introduces new files in the common include/linux/pds
include directory.

Changes:
v2:
- Implement state transitions for VFIO_MIGRATION_P2P flag
- Improve auxiliary driver probe by returning EPROBE_DEFER
when the PCI driver is not set up correctly
- Add pointer to docs in
Documentation/networking/device_drivers/ethernet/index.rst

Brett Creeley (7):
pds_vfio: Initial support for pds_vfio VFIO driver
pds_vfio: Add support to register as PDS client
pds_vfio: Add VFIO live migration support
vfio: Commonize combine_ranges for use in other VFIO drivers
pds_vfio: Add support for dirty page tracking
pds_vfio: Add support for firmware recovery
pds_vfio: Add documentation files

.../ethernet/pensando/pds_vfio.rst | 88 +++
drivers/vfio/pci/Kconfig | 2 +
drivers/vfio/pci/mlx5/cmd.c | 48 +-
drivers/vfio/pci/pds/Kconfig | 10 +
drivers/vfio/pci/pds/Makefile | 12 +
drivers/vfio/pci/pds/aux_drv.c | 216 +++++++
drivers/vfio/pci/pds/aux_drv.h | 30 +
drivers/vfio/pci/pds/cmds.c | 486 ++++++++++++++++
drivers/vfio/pci/pds/cmds.h | 44 ++
drivers/vfio/pci/pds/dirty.c | 541 ++++++++++++++++++
drivers/vfio/pci/pds/dirty.h | 49 ++
drivers/vfio/pci/pds/lm.c | 484 ++++++++++++++++
drivers/vfio/pci/pds/lm.h | 53 ++
drivers/vfio/pci/pds/pci_drv.c | 134 +++++
drivers/vfio/pci/pds/pci_drv.h | 9 +
drivers/vfio/pci/pds/vfio_dev.c | 238 ++++++++
drivers/vfio/pci/pds/vfio_dev.h | 42 ++
drivers/vfio/vfio_main.c | 48 ++
include/linux/pds/pds_core_if.h | 1 +
include/linux/pds/pds_lm.h | 356 ++++++++++++
include/linux/vfio.h | 3 +
21 files changed, 2847 insertions(+), 47 deletions(-)
create mode 100644 Documentation/networking/device_drivers/ethernet/pensando/pds_vfio.rst
create mode 100644 drivers/vfio/pci/pds/Kconfig
create mode 100644 drivers/vfio/pci/pds/Makefile
create mode 100644 drivers/vfio/pci/pds/aux_drv.c
create mode 100644 drivers/vfio/pci/pds/aux_drv.h
create mode 100644 drivers/vfio/pci/pds/cmds.c
create mode 100644 drivers/vfio/pci/pds/cmds.h
create mode 100644 drivers/vfio/pci/pds/dirty.c
create mode 100644 drivers/vfio/pci/pds/dirty.h
create mode 100644 drivers/vfio/pci/pds/lm.c
create mode 100644 drivers/vfio/pci/pds/lm.h
create mode 100644 drivers/vfio/pci/pds/pci_drv.c
create mode 100644 drivers/vfio/pci/pds/pci_drv.h
create mode 100644 drivers/vfio/pci/pds/vfio_dev.c
create mode 100644 drivers/vfio/pci/pds/vfio_dev.h
create mode 100644 include/linux/pds/pds_lm.h

--
2.17.1