Re: [PATCH V1 vfio 0/7] Enhance the vfio-virtio driver to support live migration

[Alex Williamson <alex.williamson@redhat.com>]

On Wed, 6 Nov 2024 04:32:31 -0500
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Mon, Nov 04, 2024 at 12:21:24PM +0200, Yishai Hadas wrote:
> > This series enhances the vfio-virtio driver to support live migration
> > for virtio-net Virtual Functions (VFs) that are migration-capable.
> >  
> > This series follows the Virtio 1.4 specification to implement the
> > necessary device parts commands, enabling a device to participate in the
> > live migration process.
> > 
> > The key VFIO features implemented include: VFIO_MIGRATION_STOP_COPY,
> > VFIO_MIGRATION_P2P, VFIO_MIGRATION_PRE_COPY.
> >  
> > The implementation integrates with the VFIO subsystem via vfio_pci_core
> > and incorporates Virtio-specific logic to handle the migration process.
> >  
> > Migration functionality follows the definitions in uapi/vfio.h and uses
> > the Virtio VF-to-PF admin queue command channel for executing the device
> > parts related commands.  
> 
> 
> virtio things here:
> 
> Acked-by: Michael S. Tsirkin <mst@redhat.com>
> 
> Alex, your tree I presume? I hope the virtio changes do not
> cause conflicts.

Sure, I can ultimately take it through my tree once we have consensus.

Yishai, please add Michael's ack to 1-4 on the next round.  Thanks,

Alex

> > Patch Overview:
> > The first four patches focus on the Virtio layer and address the
> > following:
> > - Define the layout of the device parts commands required as part of the
> >   migration process.
> > - Provide APIs to enable upper layers (e.g., VFIO, net) to execute the
> >   related device parts commands.
> >  
> > The last three patches focus on the VFIO layer:
> > - Extend the vfio-virtio driver to support live migration for Virtio-net
> >   VFs.
> > - Move legacy I/O operations to a separate file, which is compiled only
> >   when VIRTIO_PCI_ADMIN_LEGACY is configured, ensuring that live
> >   migration depends solely on VIRTIO_PCI.
> >  
> > Additional Notes:
> > - The kernel protocol between the source and target devices includes a
> >   header containing metadata such as record size, tag, and flags.
> >   The record size allows the target to read a complete image from the
> >   source before passing device part data. This follows the Virtio
> >   specification, which mandates that partial device parts are not
> >   supplied. The tag and flags serve as placeholders for future extensions
> >   to the kernel protocol between the source and target, ensuring backward
> >   and forward compatibility.
> >  
> > - Both the source and target comply with the Virtio specification by
> >   using a device part object with a unique ID during the migration
> >   process. As this resource is limited to a maximum of 255, its lifecycle
> >   is confined to periods when live migration is active.
> > 
> > - According to the Virtio specification, a device has only two states:
> >   RUNNING and STOPPED. Consequently, certain VFIO transitions (e.g.,
> >   RUNNING_P2P->STOP, STOP->RUNNING_P2P) are treated as no-ops. When
> >   transitioning to RUNNING_P2P, the device state is set to STOP and
> >   remains STOPPED until it transitions back from RUNNING_P2P->RUNNING, at
> >   which point it resumes its RUNNING state. During transition to STOP,
> >   the virtio device only stops initiating outgoing requests(e.g. DMA,
> >   MSIx, etc.) but still must accept incoming operations.
> > 
> > - Furthermore, the Virtio specification does not support reading partial
> >   or incremental device contexts. This means that during the PRE_COPY
> >   state, the vfio-virtio driver reads the full device state. This step is
> >   beneficial because it allows the device to send some "initial data"
> >   before moving to the STOP_COPY state, thus reducing downtime by
> >   preparing early and warming-up. As the device state can be changed and
> >   the benefit is highest when the pre copy data closely matches the final
> >   data we read it in a rate limiter mode and reporting no data available
> >   for some time interval after the previous call. With PRE_COPY enabled,
> >   we observed a downtime reduction of approximately 70-75% in various
> >   scenarios compared to when PRE_COPY was disabled, while keeping the
> >   total migration time nearly the same.
> > 
> > - Support for dirty page tracking during migration will be provided via
> >   the IOMMUFD framework.
> >  
> > - This series has been successfully tested on Virtio-net VF devices.
> > 
> > Changes from V0:
> > https://lore.kernel.org/kvm/20241101102518.1bf2c6e6.alex.williamson@redhat.com/T/
> > 
> > Vfio:
> > Patch #5:
> > - Enhance the commit log to provide a clearer explanation of P2P
> >   behavior over Virtio devices, as discussed on the mailing list.
> > Patch #6:
> > - Implement the rate limiter mechanism as part of the PRE_COPY state,
> >   following Alex’s suggestion.
> > - Update the commit log to include actual data demonstrating the impact of
> >   PRE_COPY, as requested by Alex.
> > Patch #7:
> > - Update the default driver operations (i.e., vfio_device_ops) to use
> >   the live migration set, and expand it to include the legacy I/O
> >   operations if they are compiled and supported.
> > 
> > Yishai
> > 
> > Yishai Hadas (7):
> >   virtio_pci: Introduce device parts access commands
> >   virtio: Extend the admin command to include the result size
> >   virtio: Manage device and driver capabilities via the admin commands
> >   virtio-pci: Introduce APIs to execute device parts admin commands
> >   vfio/virtio: Add support for the basic live migration functionality
> >   vfio/virtio: Add PRE_COPY support for live migration
> >   vfio/virtio: Enable live migration once VIRTIO_PCI was configured
> > 
> >  drivers/vfio/pci/virtio/Kconfig     |    4 +-
> >  drivers/vfio/pci/virtio/Makefile    |    3 +-
> >  drivers/vfio/pci/virtio/common.h    |  127 +++
> >  drivers/vfio/pci/virtio/legacy_io.c |  420 +++++++++
> >  drivers/vfio/pci/virtio/main.c      |  500 ++--------
> >  drivers/vfio/pci/virtio/migrate.c   | 1336 +++++++++++++++++++++++++++
> >  drivers/virtio/virtio_pci_common.h  |   19 +-
> >  drivers/virtio/virtio_pci_modern.c  |  457 ++++++++-
> >  include/linux/virtio.h              |    1 +
> >  include/linux/virtio_pci_admin.h    |   11 +
> >  include/uapi/linux/virtio_pci.h     |  131 +++
> >  11 files changed, 2594 insertions(+), 415 deletions(-)
> >  create mode 100644 drivers/vfio/pci/virtio/common.h
> >  create mode 100644 drivers/vfio/pci/virtio/legacy_io.c
> >  create mode 100644 drivers/vfio/pci/virtio/migrate.c
> > 
> > -- 
> > 2.27.0  
>