[PATCH v1 00/13] Multifd 🔀 device state transfer support with VFIO consumer

["Maciej S. Szmigiero" <mail@maciej.szmigiero.name>]

From: "Maciej S. Szmigiero" <maciej.szmigiero@oracle.com>

This is an updated v1 patch series of the RFC (v0) series located here:
https://lore.kernel.org/qemu-devel/cover.1713269378.git.maciej.szmigiero@oracle.com/


Changes from the RFC (v0):
* Extend the existing multifd packet format instead of introducing a new
migration channel header.

* As the replacement of switching the migration channel header on or off
introduce "x-migration-multifd-transfer" VFIO device property instead that
allows configuring at runtime whether to send the particular device state
via multifd channels when live migrating that device.

This property defaults to "false" for bit stream compatibility with older
QEMU versions.

* Remove the support for having dedicated device state transfer multifd
channels since the same downtime performance can be attained by simply
reducing the total number of multifd channels in a shared channel
configuration to the number of channels available for RAM transfer in
the dedicated device state channels configuration.

For example, the best downtime from the dedicated device state config
on my setup (achieved in configuration of 10 total multifd channels /
4 dedicated device state channels) can also be achieved in the
shared RAM/device state channel configuration by reducing the total
multifd channel count to 6.

It looks like not having too many RAM transfer multifd channels is
key to having a good downtime since the results are reproducibly
worse with 15 shared channels total, while they are as good as with
6 shared channels if there are 15 total channels but 4 of them are
dedicated to transferring device state (leaving 11 for RAM transfer).

* Make the next multifd channel selection more fair when converting
multifd_send_pages::next_channel to atomic.

* Convert the code to use QEMU thread sync primitives (QemuMutex with
QemuLockable, QemuCond) instead of their Glib equivalents (GMutex,
GMutexLocker and GCond).

* Rename complete_precopy_async{,_wait} to complete_precopy_{begin,_end} as
suggested.

* Rebase onto the last week's QEMU git master and retest.


When working on the updated patch set version I also investigated the
possibility of refactoring VM live phase (non-downtime) transfers to
happen via multifd channels.

However, the VM live phase transfer works differently: it happens
opportunistically until the remaining data drops below the switchover
threshold, rather that transferring always the whole device state data
until their exhaustion.

For this reason, there would need to be some way in the migration
framework to update the remaining data estimate from per-device
saving/transfer queuing thread and then stop these threads when the
decision has been reached in the migration core to stop the VM and
switch over. Such functionality would need to be introduced first.

There would also need to be some fairness guarantees so every device
gets similar access to multifd channels - otherwise there could be a
situation that the remaining data never drops below switchover
threshold because some devices are starved with respect to access to
the multifd transfer channels - as in the VM live phase additional
device data is constantly being generated.

Moreover, there's nothing stopping a QEMU device driver from requiring
different handling (loading, etc.) of VM live phase data from the
post-switchover data.
For cases like this some kind of a new device VM live phase incoming
data load handler would need to be introduced too.

For the above reasons, the VM live phase multifd transfer functionality
isn't a simple extension of the functionality introduced by this patch
set.


For convenience, this patch set is also available as a git tree:
https://github.com/maciejsszmigiero/qemu/tree/multifd-device-state-transfer-vfio


Maciej S. Szmigiero (13):
  vfio/migration: Add save_{iterate,complete_precopy}_started trace
    events
  migration/ram: Add load start trace event
  migration/multifd: Zero p->flags before starting filling a packet
  migration: Add save_live_complete_precopy_{begin,end} handlers
  migration: Add qemu_loadvm_load_state_buffer() and its handler
  migration: Add load_finish handler and associated functions
  migration/multifd: Device state transfer support - receive side
  migration/multifd: Convert multifd_send_pages::next_channel to atomic
  migration/multifd: Device state transfer support - send side
  migration/multifd: Add migration_has_device_state_support()
  vfio/migration: Multifd device state transfer support - receive side
  vfio/migration: Add x-migration-multifd-transfer VFIO property
  vfio/migration: Multifd device state transfer support - send side

 hw/vfio/migration.c           | 545 +++++++++++++++++++++++++++++++++-
 hw/vfio/pci.c                 |   7 +
 hw/vfio/trace-events          |  15 +-
 include/hw/vfio/vfio-common.h |  27 ++
 include/migration/misc.h      |   5 +
 include/migration/register.h  |  70 +++++
 migration/migration.c         |   6 +
 migration/migration.h         |   3 +
 migration/multifd-zlib.c      |   2 +-
 migration/multifd-zstd.c      |   2 +-
 migration/multifd.c           | 336 +++++++++++++++++----
 migration/multifd.h           |  57 +++-
 migration/ram.c               |   1 +
 migration/savevm.c            | 112 +++++++
 migration/savevm.h            |   7 +
 migration/trace-events        |   1 +
 16 files changed, 1132 insertions(+), 64 deletions(-)