Re: [RFC PATCH vfio 0/7] pds vfio driver

[Brett Creeley <bcreeley@amd.com>]

On 12/11/2022 5:16 PM, Brett Creeley wrote:
> 
> On 12/11/2022 4:54 AM, Max Gurtovoy wrote:
>> Caution: This message originated from an External Source. Use proper 
>> caution when opening attachments, clicking links, or responding.
>>
>>
>> On 12/7/2022 3:06 AM, Brett Creeley wrote:
>>> This is a first draft patchset for a new vendor specific VFIO driver for
>>> use with the AMD/Pensando Distributed Services Card (DSC). This driver
>>> (pds_vfio) is a client of the newly introduced pds_core driver.
>>>
>>> Reference to the pds_core patchset:
>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.kernel.org%2Fnetdev%2F20221207004443.33779-1-shannon.nelson%40amd.com%2F&data=05%7C01%7Cbrett.creeley%40amd.com%7C0591fe11a7c24bf8789908dadb76db84%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638063600829691750%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3VMvNtUto4HwCap65NvWrIADbLzGk4Ef9ZnA9fAB458%3D&reserved=0
>>>
>>> AMD/Pensando already supports a NVMe VF device (1dd8:1006) in the
>>> Distributed Services Card (DSC). This patchset adds the new pds_vfio
>>> driver in order to support NVMe VF live migration.
>>>
>>> This driver will use the pds_core device and auxiliary_bus as the VFIO
>>> control path to the DSC. The pds_core device creates auxiliary_bus 
>>> devices
>>> for each live migratable VF. The devices are named by their feature plus
>>> the VF PCI BDF so the auxiliary_bus driver implemented by pds_vfio 
>>> can find
>>> its related VF PCI driver instance. Once this auxiliary bus connection
>>> is configured, the pds_vfio driver can send admin queue commands to the
>>> device and receive events from pds_core.
>>>
>>> An ASCII diagram of a VFIO instance looks something like this and can
>>> be used with the VFIO subsystem to provide devices VFIO and live
>>> migration support.
>>>
>>>                                 .------.  .--------------------------.
>>>                                 | QEMU |--|  VM     .-------------.  |
>>>                                 '......'  |         | nvme driver |  |
>>>                                    |      |         .-------------.  |
>>>                                    |      |         |  SR-IOV VF  |  |
>>>                                    |      |         '-------------'  |
>>>                                    |      '---------------||---------'
>>>                                 .--------------.          ||
>>>                                 |/dev/<vfio_fd>|          ||
>>>                                 '--------------'          ||
>>> Host Userspace                         |                 ||
>>> ===================================================      ||
>>> Host Kernel                            |                 ||
>>>                                         |                 ||
>>>             pds_core.LM.2305 <--+   .--------.            ||
>>>                     |           |   |vfio-pci|            ||
>>>                     |           |   '--------'            ||
>>>                     |           |       |                 ||
>>>           .------------.       .-------------.            ||
>>>           |  pds_core  |       |   pds_vfio  |            ||
>>>           '------------'       '-------------'            ||
>>>                 ||                   ||                   ||
>>>               09:00.0              09:00.1                ||
>>> == PCI ==================================================||=====
>>>                 ||                   ||                   ||
>>>            .----------.         .----------.              ||
>>>      ,-----|    PF    |---------|    VF    |-------------------,
>>>      |     '----------'         '----------'  |      nvme      |
>>>      |                     DSC                |  data/control  |
>>>      |                                        |      path      |
>>>      -----------------------------------------------------------
>>
>> Hi Brett,
>>
>> what is the class code of the pds_core device ?
>>
>> I see that pds_vfio class_code is PCI_CLASS_STORAGE_EXPRESS.
> 
> The pds_core driver has the following as its only pci_device_id
> entry:
> 
> PCI_VDEVICE(PENSANDO, PCI_DEVICE_ID_PENSANDO_CORE_PF)

The PCI class code for this device is 0x12 (Processing accelerator).

Thanks,

Brett
> 
>>
>>>
>>>
>>> The pds_vfio driver is targeted to reside in drivers/vfio/pci/pds.
>>> It makes use of and introduces new files in the common include/linux/pds
>>> include directory.
>>>
>>> Brett Creeley (7):
>>>    pds_vfio: Initial support for pds_vfio VFIO driver
>>>    pds_vfio: Add support to register as PDS client
>>>    pds_vfio: Add VFIO live migration support
>>>    vfio: Commonize combine_ranges for use in other VFIO drivers
>>>    pds_vfio: Add support for dirty page tracking
>>>    pds_vfio: Add support for firmware recovery
>>>    pds_vfio: Add documentation files
>>>
>>>   .../ethernet/pensando/pds_vfio.rst            |  88 +++
>>>   drivers/vfio/pci/Kconfig                      |   2 +
>>>   drivers/vfio/pci/mlx5/cmd.c                   |  48 +-
>>>   drivers/vfio/pci/pds/Kconfig                  |  10 +
>>>   drivers/vfio/pci/pds/Makefile                 |  12 +
>>>   drivers/vfio/pci/pds/aux_drv.c                | 216 +++++++
>>>   drivers/vfio/pci/pds/aux_drv.h                |  30 +
>>>   drivers/vfio/pci/pds/cmds.c                   | 486 ++++++++++++++++
>>>   drivers/vfio/pci/pds/cmds.h                   |  44 ++
>>>   drivers/vfio/pci/pds/dirty.c                  | 541 ++++++++++++++++++
>>>   drivers/vfio/pci/pds/dirty.h                  |  49 ++
>>>   drivers/vfio/pci/pds/lm.c                     | 484 ++++++++++++++++
>>>   drivers/vfio/pci/pds/lm.h                     |  53 ++
>>>   drivers/vfio/pci/pds/pci_drv.c                | 134 +++++
>>>   drivers/vfio/pci/pds/pci_drv.h                |   9 +
>>>   drivers/vfio/pci/pds/vfio_dev.c               | 238 ++++++++
>>>   drivers/vfio/pci/pds/vfio_dev.h               |  42 ++
>>>   drivers/vfio/vfio_main.c                      |  48 ++
>>>   include/linux/pds/pds_core_if.h               |   1 +
>>>   include/linux/pds/pds_lm.h                    | 356 ++++++++++++
>>>   include/linux/vfio.h                          |   3 +
>>>   21 files changed, 2847 insertions(+), 47 deletions(-)
>>>   create mode 100644 
>>> Documentation/networking/device_drivers/ethernet/pensando/pds_vfio.rst
>>>   create mode 100644 drivers/vfio/pci/pds/Kconfig
>>>   create mode 100644 drivers/vfio/pci/pds/Makefile
>>>   create mode 100644 drivers/vfio/pci/pds/aux_drv.c
>>>   create mode 100644 drivers/vfio/pci/pds/aux_drv.h
>>>   create mode 100644 drivers/vfio/pci/pds/cmds.c
>>>   create mode 100644 drivers/vfio/pci/pds/cmds.h
>>>   create mode 100644 drivers/vfio/pci/pds/dirty.c
>>>   create mode 100644 drivers/vfio/pci/pds/dirty.h
>>>   create mode 100644 drivers/vfio/pci/pds/lm.c
>>>   create mode 100644 drivers/vfio/pci/pds/lm.h
>>>   create mode 100644 drivers/vfio/pci/pds/pci_drv.c
>>>   create mode 100644 drivers/vfio/pci/pds/pci_drv.h
>>>   create mode 100644 drivers/vfio/pci/pds/vfio_dev.c
>>>   create mode 100644 drivers/vfio/pci/pds/vfio_dev.h
>>>   create mode 100644 include/linux/pds/pds_lm.h
>>>
>>> -- 
>>> 2.17.1
>>>