Re: [PATCH V1 vfio 0/7] Enhance the vfio-virtio driver to support live migration

["Michael S. Tsirkin" <mst@redhat.com>]

On Mon, Nov 04, 2024 at 12:21:24PM +0200, Yishai Hadas wrote:
> This series enhances the vfio-virtio driver to support live migration
> for virtio-net Virtual Functions (VFs) that are migration-capable.
>  
> This series follows the Virtio 1.4 specification to implement the
> necessary device parts commands, enabling a device to participate in the
> live migration process.
> 
> The key VFIO features implemented include: VFIO_MIGRATION_STOP_COPY,
> VFIO_MIGRATION_P2P, VFIO_MIGRATION_PRE_COPY.
>  
> The implementation integrates with the VFIO subsystem via vfio_pci_core
> and incorporates Virtio-specific logic to handle the migration process.
>  
> Migration functionality follows the definitions in uapi/vfio.h and uses
> the Virtio VF-to-PF admin queue command channel for executing the device
> parts related commands.


virtio things here:

Acked-by: Michael S. Tsirkin <mst@redhat.com>

Alex, your tree I presume? I hope the virtio changes do not
cause conflicts.


> Patch Overview:
> The first four patches focus on the Virtio layer and address the
> following:
> - Define the layout of the device parts commands required as part of the
>   migration process.
> - Provide APIs to enable upper layers (e.g., VFIO, net) to execute the
>   related device parts commands.
>  
> The last three patches focus on the VFIO layer:
> - Extend the vfio-virtio driver to support live migration for Virtio-net
>   VFs.
> - Move legacy I/O operations to a separate file, which is compiled only
>   when VIRTIO_PCI_ADMIN_LEGACY is configured, ensuring that live
>   migration depends solely on VIRTIO_PCI.
>  
> Additional Notes:
> - The kernel protocol between the source and target devices includes a
>   header containing metadata such as record size, tag, and flags.
>   The record size allows the target to read a complete image from the
>   source before passing device part data. This follows the Virtio
>   specification, which mandates that partial device parts are not
>   supplied. The tag and flags serve as placeholders for future extensions
>   to the kernel protocol between the source and target, ensuring backward
>   and forward compatibility.
>  
> - Both the source and target comply with the Virtio specification by
>   using a device part object with a unique ID during the migration
>   process. As this resource is limited to a maximum of 255, its lifecycle
>   is confined to periods when live migration is active.
> 
> - According to the Virtio specification, a device has only two states:
>   RUNNING and STOPPED. Consequently, certain VFIO transitions (e.g.,
>   RUNNING_P2P->STOP, STOP->RUNNING_P2P) are treated as no-ops. When
>   transitioning to RUNNING_P2P, the device state is set to STOP and
>   remains STOPPED until it transitions back from RUNNING_P2P->RUNNING, at
>   which point it resumes its RUNNING state. During transition to STOP,
>   the virtio device only stops initiating outgoing requests(e.g. DMA,
>   MSIx, etc.) but still must accept incoming operations.
> 
> - Furthermore, the Virtio specification does not support reading partial
>   or incremental device contexts. This means that during the PRE_COPY
>   state, the vfio-virtio driver reads the full device state. This step is
>   beneficial because it allows the device to send some "initial data"
>   before moving to the STOP_COPY state, thus reducing downtime by
>   preparing early and warming-up. As the device state can be changed and
>   the benefit is highest when the pre copy data closely matches the final
>   data we read it in a rate limiter mode and reporting no data available
>   for some time interval after the previous call. With PRE_COPY enabled,
>   we observed a downtime reduction of approximately 70-75% in various
>   scenarios compared to when PRE_COPY was disabled, while keeping the
>   total migration time nearly the same.
> 
> - Support for dirty page tracking during migration will be provided via
>   the IOMMUFD framework.
>  
> - This series has been successfully tested on Virtio-net VF devices.
> 
> Changes from V0:
> https://lore.kernel.org/kvm/20241101102518.1bf2c6e6.alex.williamson@redhat.com/T/
> 
> Vfio:
> Patch #5:
> - Enhance the commit log to provide a clearer explanation of P2P
>   behavior over Virtio devices, as discussed on the mailing list.
> Patch #6:
> - Implement the rate limiter mechanism as part of the PRE_COPY state,
>   following Alex’s suggestion.
> - Update the commit log to include actual data demonstrating the impact of
>   PRE_COPY, as requested by Alex.
> Patch #7:
> - Update the default driver operations (i.e., vfio_device_ops) to use
>   the live migration set, and expand it to include the legacy I/O
>   operations if they are compiled and supported.
> 
> Yishai
> 
> Yishai Hadas (7):
>   virtio_pci: Introduce device parts access commands
>   virtio: Extend the admin command to include the result size
>   virtio: Manage device and driver capabilities via the admin commands
>   virtio-pci: Introduce APIs to execute device parts admin commands
>   vfio/virtio: Add support for the basic live migration functionality
>   vfio/virtio: Add PRE_COPY support for live migration
>   vfio/virtio: Enable live migration once VIRTIO_PCI was configured
> 
>  drivers/vfio/pci/virtio/Kconfig     |    4 +-
>  drivers/vfio/pci/virtio/Makefile    |    3 +-
>  drivers/vfio/pci/virtio/common.h    |  127 +++
>  drivers/vfio/pci/virtio/legacy_io.c |  420 +++++++++
>  drivers/vfio/pci/virtio/main.c      |  500 ++--------
>  drivers/vfio/pci/virtio/migrate.c   | 1336 +++++++++++++++++++++++++++
>  drivers/virtio/virtio_pci_common.h  |   19 +-
>  drivers/virtio/virtio_pci_modern.c  |  457 ++++++++-
>  include/linux/virtio.h              |    1 +
>  include/linux/virtio_pci_admin.h    |   11 +
>  include/uapi/linux/virtio_pci.h     |  131 +++
>  11 files changed, 2594 insertions(+), 415 deletions(-)
>  create mode 100644 drivers/vfio/pci/virtio/common.h
>  create mode 100644 drivers/vfio/pci/virtio/legacy_io.c
>  create mode 100644 drivers/vfio/pci/virtio/migrate.c
> 
> -- 
> 2.27.0