[RFC PATCH v2 9/9] docs: Add igb VF migration testing setup guide

["Cédric Le Goater" <clg@redhat.com>]

Document the igb VF migration interface: DVSEC register layout, dirty
page tracking, testing setup with nested virtualization.

AI-used-for: docs
Signed-off-by: Cédric Le Goater <clg@redhat.com>
---
 MAINTAINERS                           |   1 +
 docs/system/device-emulation.rst      |   1 +
 docs/system/devices/igb-migration.rst | 417 ++++++++++++++++++++++++++
 docs/system/devices/igb.rst           |   6 +
 4 files changed, 425 insertions(+)
 create mode 100644 docs/system/devices/igb-migration.rst

diff --git a/MAINTAINERS b/MAINTAINERS
index f88b526be238..4a2f1357e936 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2820,6 +2820,7 @@ igb VF migration
 M: Cédric Le Goater <clg@redhat.com>
 S: Maintained
 F: hw/net/igb_migration.*
+F: docs/system/devices/igb-migration.rst
 
 eepro100
 M: Stefan Weil <sw@weilnetz.de>
diff --git a/docs/system/device-emulation.rst b/docs/system/device-emulation.rst
index 40054bb7dfcc..75f423b795cd 100644
--- a/docs/system/device-emulation.rst
+++ b/docs/system/device-emulation.rst
@@ -90,6 +90,7 @@ Emulated Devices
    devices/cxl.rst
    devices/emmc.rst
    devices/igb.rst
+   devices/igb-migration.rst
    devices/ivshmem-flat.rst
    devices/ivshmem.rst
    devices/keyboard.rst
diff --git a/docs/system/devices/igb-migration.rst b/docs/system/devices/igb-migration.rst
new file mode 100644
index 000000000000..d72f14f5fe63
--- /dev/null
+++ b/docs/system/devices/igb-migration.rst
@@ -0,0 +1,417 @@
+.. SPDX-License-Identifier: GPL-2.0-or-later
+.. _igb-migration:
+
+igb VF Migration
+----------------
+
+Live migration of VFIO-passthrough devices (SR-IOV VFs, vGPUs) is a
+growing requirement, but real hardware with migration support is scarce
+and hard to debug. An emulated device provides a fully controlled
+testbed for developing and validating the entire software stack --
+vfio-pci variant drivers, VFIO core migration v2 framework, QEMU,
+libvirt -- and for tuning complex migration policies such as downtime
+convergence. It also serves as an educational reference for
+understanding VFIO migration end-to-end, from device state
+serialization to dirty page tracking.
+
+The igb device supports an experimental VF migration interface that allows
+the `igb-vfio-pci`_ variant driver to migrate VF state during live
+migration using the standard VFIO migration v2 protocol with stop-copy
+and pre-copy support.
+
+This is enabled with the ``x-vf-migration`` property::
+
+  -device igb,x-vf-migration=on,...
+
+Each emulated VF then advertises a DVSEC discovered by the
+`igb-vfio-pci`_ variant driver at bind time. This feature is
+experimental (``x-`` prefix, default off).
+
+Architecture
+~~~~~~~~~~~~
+
+The target scenario is nested virtualization::
+
+  L0 QEMU
+    igb PF with x-vf-migration=on
+    └── VFs with migration DVSEC
+
+  L1 kernel
+    igb-vfio-pci variant driver
+    translates VFIO migration v2 ioctls → DVSEC config writes
+
+  L1 QEMU (stock, unmodified)
+    vfio-pci device model, standard migration fd
+
+  L2 guest
+    standard igbvf driver, unaware of migration
+
+The L1 QEMU is completely unmodified -- it sees a standard VFIO
+migratable device and uses the normal migration fd path. The
+`igb-vfio-pci`_ variant driver handles the translation between
+VFIO migration v2 ioctls and DVSEC config writes.
+
+Design
+~~~~~~
+
+The migration interface is exposed through a DVSEC at offset 0x160
+in VF extended config space (see `DVSEC register layout`_ below for
+the full register map).
+
+Device state is serialized as a versioned blob of per-VF register
+(offset, value) pairs covering control, interrupt, RX/TX queue,
+receive address (RA/RA2), etc. plus TX context descriptors and
+VFRE/VFTE enable bits. The buffer address is a guest physical address
+(GPA) written by the driver via ``virt_to_phys``; the device accesses
+guest RAM directly through the system address space.
+
+Dirty page tracking is implemented with per-range bitmaps maintained
+in IGBCore. All VF DMA paths in ``igb_core.c`` (TX data, RX data,
+descriptor writeback) are instrumented to record touched pages. The
+`igb-vfio-pci`_ variant driver registers tracked IOVA ranges and
+queries dirty bitmaps through a shared buffer. Buffer structures
+include len, flags, and reserved fields for future extensibility.
+
+The dirty bitmaps are maintained inside the device, which is not
+realistic for discrete NICs without on-chip DRAM.
+
+DVSEC register layout
+~~~~~~~~~~~~~~~~~~~~~
+
+The migration DVSEC (36 bytes at offset ``0x160``) uses a command
+doorbell model. All commands are synchronous -- the device completes
+the operation before the config write returns::
+
+  Offset  Name          Access  Description
+  +0x00   ExtCap Hdr    RO      PCIe extended cap (id=0x23, ver=1)
+  +0x04   DVSEC Hdr 1   RO      length[31:20] | rev[19:16] | vendor_id[15:0]
+  +0x08   DVSEC Hdr 2   RO      DVSEC ID (1)
+  +0x0A   Reserved      -       Padding for DWORD alignment
+  +0x0C   CAPS          RO      F_STATE[0], F_DIRTY[1], max_ranges[11:8], pgsize[16:12]
+  +0x10   CTRL          WO      Doorbell: cmd[7:0], arg[31:8]
+  +0x14   STATUS        RO      state[7:0], error_code[15:8], quiesced[16]
+  +0x18   BUF_ADDR_LO   RW      Shared DMA buffer GPA (low 32 bits)
+  +0x1C   BUF_ADDR_HI   RW      Shared DMA buffer GPA (high 32 bits)
+  +0x20   DATA_SIZE     RO      State blob size in bytes
+
+CTRL commands::
+
+  Cmd  Name            Arg             Description
+  1    SET_STATE       state[31:8]     Set migration state
+  2    SAVE            -               DMA-write state to buffer
+  3    LOAD            size[31:8]      DMA-read state from buffer
+  4    DIRTY_ENABLE    -               Enable dirty tracking (params in DMA buffer)
+  5    DIRTY_DISABLE   -               Disable dirty tracking
+  6    DIRTY_QUERY     -               Query dirty bitmap (via DMA buffer)
+  7    GET_STATS       -               Query statistics (via DMA buffer)
+
+The driver sets ``BUF_ADDR_LO/HI`` before issuing commands that use a
+DMA buffer (SAVE, LOAD, DIRTY_ENABLE, DIRTY_QUERY, GET_STATS). The
+buffer address is latched per CTRL write, so the driver can use
+different buffers for different commands. The buffer address is a
+guest physical address (GPA).
+
+State transitions follow the VFIO migration v2 state machine. The
+driver issues ``SET_STATE`` with the target state in the arg field and
+reads ``STATUS`` to confirm the transition. Device states::
+
+  0  ERROR       1  STOP       2  RUNNING
+  3  STOP_COPY   4  RESUMING   5  PRE_COPY
+
+``DATA_SIZE`` reflects the state blob size. At reset and in ``STOP``
+state it holds the maximum size the driver should allocate. After
+``SET_STATE(STOP_COPY)`` or ``SAVE`` it holds the actual serialized
+size. The driver reads it after entering ``STOP_COPY`` to allocate an
+exact-sized DMA buffer before issuing ``SAVE``.
+
+The state blob is a versioned sequence of register (offset, value)
+pairs with magic ``0x4D494742`` ("MIGB").
+
+When ``STATUS`` state is ``ERROR`` (0), bits [15:8] contain an error
+code identifying the failure::
+
+  1   UNK_CMD           Unknown CTRL command
+  2   BAD_STATE         Command issued in wrong migration state
+  3   NO_BUFFER         Command requires buffer but BUF_ADDR not set
+  4   DMA_FAILED        DMA transfer to/from buffer failed
+  5   BAD_SIZE          State blob too large or empty
+  6   BAD_MAGIC         State blob magic mismatch
+  7   BAD_VERSION       State blob version mismatch
+  8   TOO_MANY_RANGES   Exceeds max_ranges from CAPS
+  9   BAD_RANGE         Invalid range (zero size, misaligned, not contained)
+  10  BAD_PGSIZE        Invalid or misaligned page size
+  11  NOT_ENABLED       Dirty query without prior enable
+
+Dirty page tracking
+~~~~~~~~~~~~~~~~~~~
+
+The migration interface supports per-VF dirty page tracking, advertised
+by the ``F_DIRTY`` flag (bit 1) in ``CAPS``. This allows the variant
+driver to enter ``PRE_COPY`` state while the VM continues to run,
+iterating on dirty pages to reduce the final stop-and-copy window.
+
+The device maintains one dirty tracking engine per range, each with its
+own bitmap scoped to the range boundaries. The ``CAPS`` register
+advertises the maximum number of ranges in bits [11:8].
+
+Dirty tracking is controlled through CTRL commands:
+
+- **DIRTY_ENABLE** (4): the driver fills an ``igb_mig_dirty_enable_req``
+  struct in the DMA buffer with the page size, range IOVA, and range
+  size, then issues the command. The device allocates a bitmap for the
+  range and begins recording pages touched by DMA. Supported page sizes
+  are advertised in ``CAPS`` bits [16:12] (bit N = 2^N bytes). The driver
+  checks ``STATUS`` for errors after the command completes.
+- **DIRTY_DISABLE** (5): tears down all ranges and stops tracking.
+- **DIRTY_QUERY** (6): the driver writes (iova, size) into the
+  ``igb_mig_dirty_query`` DMA buffer, then issues the command. The
+  device validates the range, copies the dirty bitmap into the buffer,
+  and clears the tracked bits after successful DMA. The driver checks
+  ``STATUS`` for errors after the command.
+
+Dirty enable DMA buffer
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``DIRTY_ENABLE`` command reads its parameters from the DMA buffer.
+The ``len`` field holds the total structure size (including reserved
+bytes) so the device can detect newer formats. ``flags`` and
+``reserved`` must be zero::
+
+  Offset  Field        Type      Description
+  0x00    len          uint32    Structure size in bytes
+  0x04    flags        uint32    Reserved, must be 0
+  0x08    pgsize       uint64    Page granularity (must match a CAPS pgsize bit)
+  0x10    range_iova   uint64    Tracked range start address
+  0x18    range_size   uint64    Tracked range size in bytes
+  0x20    reserved[4]  uint32    Reserved, must be 0
+
+Dirty query DMA buffer
+~~~~~~~~~~~~~~~~~~~~~~
+
+The ``DIRTY_QUERY`` command uses a shared DMA buffer for both request
+and response. The ``len`` field holds the total buffer size (header +
+bitmap). ``flags`` and ``reserved`` must be zero::
+
+  Offset  Field              Written by  Description
+  0x00    len                driver      Total buffer size in bytes
+  0x04    flags              driver      Reserved, must be 0
+  0x08    iova               driver      Query range start
+  0x10    size               driver      Query range size
+  0x18    bitmap_size        device      Bytes written to bitmap
+  0x1C    dirty_page_count   device      Number of set bits
+  0x20    dma_writes         device      DMA write count (diagnostic)
+  0x28    reserved[6]        -           Reserved, must be 0
+  0x40    bitmap[]           device      Dirty page bitmap
+
+Migration statistics
+~~~~~~~~~~~~~~~~~~~~
+
+The ``GET_STATS`` command DMA-writes a statistics response into the
+driver-provided buffer. The driver sets ``BUF_ADDR_LO/HI`` and issues
+the command; the device writes the response and returns::
+
+  Offset  Field                Type      Description
+  0x00    dma_writes           uint64    DMA write operations tracked
+  0x08    dma_bytes            uint64    DMA bytes written
+  0x10    dirty_pages_set      uint32    Dirty pages marked since enable
+  0x14    dirty_pages_cleared  uint32    Dirty pages cleared by queries
+  0x18    dirty_page_count     uint32    Current dirty pages (set - cleared)
+  0x1C    dirty_query_count    uint32    Number of QUERY operations
+
+The variant driver exposes these via debugfs at
+``/sys/kernel/debug/vfio/<device>/migration/dirty/stats``.
+
+Testing setup
+~~~~~~~~~~~~~
+
+The target scenario is nested virtualization: L0 runs QEMU with an
+igb PF (``x-vf-migration=on``), L1 runs the `igb-vfio-pci`_ variant
+driver and an unmodified QEMU, and L2 runs a standard igbvf driver.
+See `Architecture`_ above for the full stack diagram.
+
+NetworkManager configuration
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In a nested setup, the L1 VMs (source and destination) have emulated
+igb PFs connected to the L0 bridge. By default, NetworkManager
+acquires DHCP leases on those PF interfaces and on any igb VFs created
+later. This causes the VF MAC address to be learned on the L0 bridge,
+which can misdirect iperf3 traffic after migration.
+
+To prevent this, configure NetworkManager on **both L1 VMs** and on the
+**L2 guest disk image**.
+
+L1 VMs (source and destination)
+...............................
+
+1. Prevent NetworkManager from managing igbvf interfaces:
+
+.. code-block:: bash
+
+   cat > /etc/NetworkManager/conf.d/99-no-igbvf.conf <<EOF
+   [keyfile]
+   unmanaged-devices=driver:igbvf
+   EOF
+
+2. Disable IP on the igb PF connections (keep the interfaces UP for
+   bridging, but with no DHCP lease):
+
+.. code-block:: bash
+
+   # Identify the NM connections for the igb PFs (NOT the virtio management NIC)
+   nmcli -t -f NAME,DEVICE connection show
+
+   # For each igb PF connection:
+   nmcli connection modify "<igb-pf-connection>" ipv4.method disabled ipv6.method disabled
+
+3. Reload NetworkManager:
+
+.. code-block:: bash
+
+   nmcli general reload
+
+L2 guest disk image
+...................
+
+Use ``virt-customize`` to add the igbvf unmanaged config to the guest
+image (offline, before any test run):
+
+.. code-block:: bash
+
+   virt-customize -a /srv/migration/rhel10.qcow2 \
+     --write /etc/NetworkManager/conf.d/99-no-igbvf.conf:'[keyfile]
+   unmanaged-devices=driver:igbvf'
+
+Network diagram
+^^^^^^^^^^^^^^^
+
+The diagram below shows the nested setup where the source and
+destination hosts are themselves VMs (L1) running on a physical
+host (L0) that emulates the igb NIC::
+
+  ┌────────────────────────────────────────────────────────────────────────────┐
+  │  L0: physical host                                                         │
+  │                                                                            │
+  │  virbr0  192.168.199.1/24                                                  │
+  │  ├── NFS server: /srv/migration                                            │
+  │  └── iperf3 client: iperf3 -c 192.168.199.200 -t 60 -i 1                   │
+  │      │                                                                     │
+  │      │  L0 virbr0 bridge (192.168.199.0/24)                                │
+  │  ────┼──────────┬──────────────────┬──────────────────────────────         │
+  │      │          │                  │                                       │
+  │      │     ┌────┴────┐        ┌────┴────┐                                  │
+  │      │     │ virtio  │        │ emulated│                                  │
+  │      │     │c0:ff:ee:│        │  igb PF │    L0 QEMU (vm6)                 │
+  │      │     │ :00:06  │        │ + igbvf │    tracks DMA dirty pages        │
+  │      │     └────┬────┘        └────┬────┘                                  │
+  │      │          │                  │                                       │
+  │  ┌───┼──────────┼──────────────────┼──────────────────────────────────┐    │
+  │  │   │  L1: vm6 (source)           │                                  │    │
+  │  │   │  enp1s0: 192.168.199.6      │                                  │    │
+  │  │   │  (management)               │                                  │    │
+  │  │   │                        enp8s0 (igb PF, no IP)                  │    │
+  │  │   │                             │                                  │    │
+  │  │   │                        igb VF0 ──► igb-vfio-pci (VFIO)         │    │
+  │  │   │                             │      dirty_sync → L0 igbvf       │    │
+  │  │   │                             │                                  │    │
+  │  │   │   virbr0                    │ VFIO passthrough                 │    │
+  │  │   │   192.168.200.1/24          │                                  │    │
+  │  │   │       │                     │                                  │    │
+  │  │   │  ┌────┼─────────────────────┼───────────────────────────┐      │    │
+  │  │   │  │    │  L2: rhel10 guest   │                           │      │    │
+  │  │   │  │    │                     │                           │      │    │
+  │  │   │  │  virtio NIC           igb VF (enp7s0)                │      │    │
+  │  │   │  │  192.168.200.130/24   192.168.199.200/24             │      │    │
+  │  │   │  │  (SSH login)          (iperf3 data path)             │      │    │
+  │  │   │  │                          │                           │      │    │
+  │  │   │  │            iperf3 -s -D  │ (listens on 0.0.0.0)      │      │    │
+  │  │   │  └──────────────────────────┼───────────────────────────┘      │    │
+  │  │   │                             │                                  │    │
+  │  │   │  virsh migrate --live ──────┼──────────────────► vm7           │    │
+  │  │   │                             │                                  │    │
+  │  └───┼─────────────────────────────┼──────────────────────────────────┘    │
+  │      │                             │                                       │
+  │      │          iperf3 traffic     │                                       │
+  │      └─────────────────────────────┘                                       │
+  │                                                                            │
+  │  ────────────────────────────────────────────────────────────────          │
+  │      │                  │                                                  │
+  │      │     ┌────────┐   │   ┌─────────┐                                    │
+  │      │     │ virtio │   │   │emulated │    L0 QEMU (vm7)                   │
+  │      │     │c0:ff:ee│   │   │ igb PF  │                                    │
+  │      │     │ :00:07 │   │   │ + igbvf │                                    │
+  │      │     └────┬───┘   │   └────┬────┘                                    │
+  │  ┌──────────────┼───────┼────────┼────────────────────────────────────┐    │
+  │  │   L1: vm7 (destination)       │                                    │    │
+  │  │   enp1s0: 192.168.199.7       │                                    │    │
+  │  │   (management)           enp8s0 (igb PF, no IP)                    │    │
+  │  │                               │                                    │    │
+  │  │                          igb VF0 ──► igb-vfio-pci (VFIO)           │    │
+  │  │                               │                                    │    │
+  │  │   virbr0                      │ VFIO passthrough                   │    │
+  │  │   192.168.200.1/24            │                                    │    │
+  │  │       │                       │                                    │    │
+  │  │  ┌────┼───────────────────────┼────────────────────────────┐       │    │
+  │  │  │    │  L2: rhel10 (after migration)                      │       │    │
+  │  │  │    │                       │                            │       │    │
+  │  │  │  virtio NIC             igb VF (enp7s0)                 │       │    │
+  │  │  │  192.168.200.130/24     192.168.199.200/24              │       │    │
+  │  │  │                            │                            │       │    │
+  │  │  │              iperf3 -s -D  │ (connection survives)      │       │    │
+  │  │  └────────────────────────────┼────────────────────────────┘       │    │
+  │  └───────────────────────────────┼────────────────────────────────────┘    │
+  │                                  │                                         │
+  │      iperf3 traffic resumes ─────┘                                         │
+  │      (same IP, same MAC, same L2 segment → transparent to client)          │
+  └────────────────────────────────────────────────────────────────────────────┘
+
+Migration under iperf3 load works correctly: dirty page tracking
+converges (from ~2000 pages per PRE_COPY iteration down to ~280 at
+STOP_COPY), and STOP_COPY stays under 250ms.
+
+Todo
+~~~~
+
+1. Add migration blocker when ``x-vf-migration=on`` (no VMState yet) or
+   add VMState support for L0 migration (dirty bitmaps, tracking
+   engines, DVSEC registers, stats)
+2. Add PRE_COPY state transfer to validate device INIT data (magic,
+   version, etc.)
+3. Add qtests for migration state machine transitions, dirty page
+   tracking
+
+Ideas
+~~~~~
+
+1. **RX bandwidth throttle** (``x-mig-rx-limit``, uint32, default 0)
+
+   Return false from ``can_receive`` when the per-VF packet count in the
+   current tracking interval exceeds the limit. Reduces DMA writes and
+   dirty pages realistically.
+
+2. **Migration phase timing** (GET_STATS extension)
+
+   Add per-VF timestamps: ``precopy_start_ns``, ``stopcopy_start_ns``,
+   ``precopy_duration_ns``, ``stopcopy_duration_ns``,
+   ``state_transition_count``. Expose via GET_STATS.
+
+3. **Hot page simulation** (``x-mig-hot-pages``, uint32, default 0)
+
+   Re-set the first N bitmap bits after each DIRTY_QUERY, simulating
+   workloads with hot pages that prevent convergence.
+
+4. **Error injection** (``x-mig-inject-error``, uint32, default 0)
+
+   One-shot error code injection before command dispatch. A separate
+   ``x-mig-inject-dma-fail`` (bool) for persistent DMA failure testing.
+
+AI disclaimer
+~~~~~~~~~~~~~
+
+Claude was used to analyze the IGB PF and VF internal state and
+identify the pain points of a working live migration of such devices.
+The generated code served as a starting point but *significant* time
+was then spent cleaning up, reworking, and shaping it into a clear,
+reviewable IGB model extension.
+
+.. _igb-vfio-pci: https://github.com/legoater/vfio-pci-extras
diff --git a/docs/system/devices/igb.rst b/docs/system/devices/igb.rst
index 50f625fd77e4..00271dbc92c3 100644
--- a/docs/system/devices/igb.rst
+++ b/docs/system/devices/igb.rst
@@ -64,6 +64,12 @@ command:
 
   pyvenv/bin/meson test --suite thorough func-x86_64-netdev_ethtool
 
+VF Migration (experimental)
+===========================
+
+See :ref:`igb-migration` for details on the experimental VF live migration
+interface.
+
 References
 ==========
 
-- 
2.55.0