[RFC PATCH 00/11] igb: Add experimental VF live migration support

["Cédric Le Goater" <clg@redhat.com>]

Hello,

Live migration of VFIO-passthrough devices - SR-IOV VFs, vGPUs - is a
growing requirement, but real hardware with migration support is
scarce and hard to debug. An emulated device provides a fully
controlled testbed for developing and validating the entire software
stack - vfio-pci variant drivers, VFIO core migration v2 framework,
QEMU, libvirt - and for tuning complex migration policies such as
downtime convergence. It also serves as an educational reference for
understanding VFIO migration end-to-end, from device state
serialization to dirty page tracking.

This series adds an experimental VF live migration interface to the
emulated igb (82576) device. It enables a vfio-pci variant driver
(igb-vfio-pci) to migrate VFs using the standard VFIO migration v2
protocol with stop-copy and pre-copy support.

The target scenario is nested virtualization:

  L0 QEMU (these patches)
    igb PF with x-vf-migration=on
    └── VFs with migration BAR + vendor cap

  L1 kernel
    igb-vfio-pci variant driver [1]
    translates VFIO migration v2 ioctls → BAR2 MMIO

  L1 QEMU (stock, unmodified)
    vfio-pci device model, standard migration fd

  L2 guest
    standard igbvf driver, unaware of migration

The L1 QEMU is completely unmodified -- it sees a standard VFIO
migratable device and uses the normal migration fd path.

* Design

The migration interface is exposed through a hidden 64KB PCI BAR
(BAR2) on each VF, discovered via a vendor-specific PCI capability
("MIGB", PCI_CAP_ID_VNDR). The BAR exposes a register-based state
machine that mirrors VFIO migration states (RUNNING, STOP, STOP_COPY,
RESUMING, PRE_COPY).

Device state is serialized as a versioned blob of per-VF register
(offset, value) pairs covering control, interrupt, RX/TX queue,
receive address (RA/RA2), etc. plus TX context descriptors and
VFRE/VFTE enable bits. The blob is transferred via DMA through the PF
device, since VFIO owns the VF's IOMMU domain and the variant driver
maps its DMA buffers through the PF.

Dirty page tracking is implemented with per-range bitmaps maintained
in IGBCore. All VF DMA paths in igb_core.c (TX data, RX data,
descriptor writeback) are instrumented to record touched pages. The
variant driver registers tracked IOVA ranges and queries dirty bitmaps
through a DMA shared buffer, using a single MMIO doorbell
(DIRTY_CTRL=QUERY) per iteration.

* Caveats

Patch 1 introduces PCI_BASE_ADDRESS_MEM_ALWAYS_ON, a QEMU-internal
BAR flag that keeps the migration BAR mapped even after VFIO's
Function Level Reset clears PCI_COMMAND_MEMORY.

The x-vf-migration property is experimental (x- prefix, default off)
and the migration BAR register interface may change.

The dirty bitmaps are maintained inside the device, which is not
realistic for discrete NICs without on-chip DRAM.

* Testing

The target scenario is nested virtualization: L0 runs QEMU with an
igb PF (x-vf-migration=on), L1 runs the igb-vfio-pci variant driver
and an unmodified QEMU, and L2 runs a standard igbvf driver.

Migration under iperf3 load works correctly: dirty page tracking
converges (from ~2000 pages per PRE_COPY iteration down to ~280 at
STOP_COPY), and STOP_COPY stays under 250ms.

* Todo/Ideas

  1. Add migration blocker when x-vf-migration=on (no VMState yet) or
     add VMState support for L0 migration (dirty bitmaps, tracking
     engines, migration BAR registers, stats)
     
  2. Add PRE_COPY match data validation (magic, version, caps)
  
  3. Support driver-provided DMA bitmaps per dirty range
     (DIRTY_RANGE_ADDR_LO/HI)
     
  4. Add QMP/HMP knobs for error injection and capability tuning:
  
    . Dirty rate throttling: artificial delay or rate limit on DMA
      dirty tracking to simulate different convergence scenarios
    . State blob size inflation: pad the migration blob to stress
      large-state transfers and test DATA_SIZE limits
    . Migration phase timing: expose per-VF counters for time spent
      in each state (PRE_COPY duration, STOP_COPY latency)
    . Hot page simulation: mark specific page ranges as always-dirty
      to test worst-case convergence
    . Error injection: force STATUS error codes, fail DMA transfers,
      corrupt state blobs, mask CAPS bits, etc.
      
  5. Add qtests for migration state machine transitions, dirty page
     tracking ?

* Credits

Alex Williamson suggested the overall approach: a hidden migration BAR
discovered via a vendor-specific PCI capability, the "vf-migration"
device property to gate the feature. Thanks for the ever ongoing
support and valuable discussions throughout these years.

* AI disclaimer

The lack of a migration-capable device has been a recurring pain point
for VFIO development over the years, and we hope this proposal
demonstrates the value of having one.

Claude was used to analyze the IGB PF and VF internal state and
identify the pain points of a working live migration of such devices.
The generated code served as a starting point but *significant* time
was then spent cleaning up, reworking, and shaping it into a clear,
reviewable proposal. As QEMU does not yet accept AI-assisted
contributions, this series is submitted as an RFC.

Thanks,

C.

[1] https://github.com/legoater/vfio-pci-extras

Cédric Le Goater (11):
  pci: Add PCI_BASE_ADDRESS_MEM_ALWAYS_ON BAR flag
  igb: Add x-vf-migration property and vendor-specific capability for
    IGBVF
  igb: Add migration BAR with state machine
  igb: Add VF state serialization for live migration
  igb: Add VF post-load fixups for live migration
  igb: Add dirty page tracking for IGBVF migration
  igb: Quiesce VFs on STOP and include PF enable state in migration blob
  igb: Fix post-migration RX ring deadlock
  igb: Send RARP after VF migration to update bridge FDB
  docs: Add igb VF migration testing setup guide
  igb: Add migration statistics registers to VF migration BAR

 MAINTAINERS                           |    6 +
 docs/system/device-emulation.rst      |    1 +
 docs/system/devices/igb-migration.rst |  313 +++++++
 docs/system/devices/igb.rst           |    6 +
 hw/net/igb_common.h                   |   11 +
 hw/net/igb_core.h                     |    8 +
 hw/net/igb_migration.h                |  201 +++++
 include/hw/pci/pci.h                  |    6 +
 hw/net/igb.c                          |   18 +
 hw/net/igb_core.c                     |  144 ++-
 hw/net/igb_migration.c                | 1194 +++++++++++++++++++++++++
 hw/net/igbvf.c                        |   29 +-
 hw/pci/pci.c                          |    6 +-
 hw/net/meson.build                    |    2 +-
 hw/net/trace-events                   |   20 +
 15 files changed, 1939 insertions(+), 26 deletions(-)
 create mode 100644 docs/system/devices/igb-migration.rst
 create mode 100644 hw/net/igb_migration.h
 create mode 100644 hw/net/igb_migration.c

-- 
2.55.0