* [RFC PATCH 00/11] igb: Add experimental VF live migration support
@ 2026-07-27 5:39 Cédric Le Goater
2026-07-27 5:39 ` [RFC PATCH 01/11] pci: Add PCI_BASE_ADDRESS_MEM_ALWAYS_ON BAR flag Cédric Le Goater
` (12 more replies)
0 siblings, 13 replies; 26+ messages in thread
From: Cédric Le Goater @ 2026-07-27 5:39 UTC (permalink / raw)
To: qemu-devel
Cc: Akihiko Odaki, Sriram Yagnaraman, Jason Wang, Alex Williamson,
Michael S . Tsirkin, Peter Xu, Avihai Horon,
Cédric Le Goater
Hello,
Live migration of VFIO-passthrough devices - SR-IOV VFs, vGPUs - is a
growing requirement, but real hardware with migration support is
scarce and hard to debug. An emulated device provides a fully
controlled testbed for developing and validating the entire software
stack - vfio-pci variant drivers, VFIO core migration v2 framework,
QEMU, libvirt - and for tuning complex migration policies such as
downtime convergence. It also serves as an educational reference for
understanding VFIO migration end-to-end, from device state
serialization to dirty page tracking.
This series adds an experimental VF live migration interface to the
emulated igb (82576) device. It enables a vfio-pci variant driver
(igb-vfio-pci) to migrate VFs using the standard VFIO migration v2
protocol with stop-copy and pre-copy support.
The target scenario is nested virtualization:
L0 QEMU (these patches)
igb PF with x-vf-migration=on
└── VFs with migration BAR + vendor cap
L1 kernel
igb-vfio-pci variant driver [1]
translates VFIO migration v2 ioctls → BAR2 MMIO
L1 QEMU (stock, unmodified)
vfio-pci device model, standard migration fd
L2 guest
standard igbvf driver, unaware of migration
The L1 QEMU is completely unmodified -- it sees a standard VFIO
migratable device and uses the normal migration fd path.
* Design
The migration interface is exposed through a hidden 64KB PCI BAR
(BAR2) on each VF, discovered via a vendor-specific PCI capability
("MIGB", PCI_CAP_ID_VNDR). The BAR exposes a register-based state
machine that mirrors VFIO migration states (RUNNING, STOP, STOP_COPY,
RESUMING, PRE_COPY).
Device state is serialized as a versioned blob of per-VF register
(offset, value) pairs covering control, interrupt, RX/TX queue,
receive address (RA/RA2), etc. plus TX context descriptors and
VFRE/VFTE enable bits. The blob is transferred via DMA through the PF
device, since VFIO owns the VF's IOMMU domain and the variant driver
maps its DMA buffers through the PF.
Dirty page tracking is implemented with per-range bitmaps maintained
in IGBCore. All VF DMA paths in igb_core.c (TX data, RX data,
descriptor writeback) are instrumented to record touched pages. The
variant driver registers tracked IOVA ranges and queries dirty bitmaps
through a DMA shared buffer, using a single MMIO doorbell
(DIRTY_CTRL=QUERY) per iteration.
* Caveats
Patch 1 introduces PCI_BASE_ADDRESS_MEM_ALWAYS_ON, a QEMU-internal
BAR flag that keeps the migration BAR mapped even after VFIO's
Function Level Reset clears PCI_COMMAND_MEMORY.
The x-vf-migration property is experimental (x- prefix, default off)
and the migration BAR register interface may change.
The dirty bitmaps are maintained inside the device, which is not
realistic for discrete NICs without on-chip DRAM.
* Testing
The target scenario is nested virtualization: L0 runs QEMU with an
igb PF (x-vf-migration=on), L1 runs the igb-vfio-pci variant driver
and an unmodified QEMU, and L2 runs a standard igbvf driver.
Migration under iperf3 load works correctly: dirty page tracking
converges (from ~2000 pages per PRE_COPY iteration down to ~280 at
STOP_COPY), and STOP_COPY stays under 250ms.
* Todo/Ideas
1. Add migration blocker when x-vf-migration=on (no VMState yet) or
add VMState support for L0 migration (dirty bitmaps, tracking
engines, migration BAR registers, stats)
2. Add PRE_COPY match data validation (magic, version, caps)
3. Support driver-provided DMA bitmaps per dirty range
(DIRTY_RANGE_ADDR_LO/HI)
4. Add QMP/HMP knobs for error injection and capability tuning:
. Dirty rate throttling: artificial delay or rate limit on DMA
dirty tracking to simulate different convergence scenarios
. State blob size inflation: pad the migration blob to stress
large-state transfers and test DATA_SIZE limits
. Migration phase timing: expose per-VF counters for time spent
in each state (PRE_COPY duration, STOP_COPY latency)
. Hot page simulation: mark specific page ranges as always-dirty
to test worst-case convergence
. Error injection: force STATUS error codes, fail DMA transfers,
corrupt state blobs, mask CAPS bits, etc.
5. Add qtests for migration state machine transitions, dirty page
tracking ?
* Credits
Alex Williamson suggested the overall approach: a hidden migration BAR
discovered via a vendor-specific PCI capability, the "vf-migration"
device property to gate the feature. Thanks for the ever ongoing
support and valuable discussions throughout these years.
* AI disclaimer
The lack of a migration-capable device has been a recurring pain point
for VFIO development over the years, and we hope this proposal
demonstrates the value of having one.
Claude was used to analyze the IGB PF and VF internal state and
identify the pain points of a working live migration of such devices.
The generated code served as a starting point but *significant* time
was then spent cleaning up, reworking, and shaping it into a clear,
reviewable proposal. As QEMU does not yet accept AI-assisted
contributions, this series is submitted as an RFC.
Thanks,
C.
[1] https://github.com/legoater/vfio-pci-extras
Cédric Le Goater (11):
pci: Add PCI_BASE_ADDRESS_MEM_ALWAYS_ON BAR flag
igb: Add x-vf-migration property and vendor-specific capability for
IGBVF
igb: Add migration BAR with state machine
igb: Add VF state serialization for live migration
igb: Add VF post-load fixups for live migration
igb: Add dirty page tracking for IGBVF migration
igb: Quiesce VFs on STOP and include PF enable state in migration blob
igb: Fix post-migration RX ring deadlock
igb: Send RARP after VF migration to update bridge FDB
docs: Add igb VF migration testing setup guide
igb: Add migration statistics registers to VF migration BAR
MAINTAINERS | 6 +
docs/system/device-emulation.rst | 1 +
docs/system/devices/igb-migration.rst | 313 +++++++
docs/system/devices/igb.rst | 6 +
hw/net/igb_common.h | 11 +
hw/net/igb_core.h | 8 +
hw/net/igb_migration.h | 201 +++++
include/hw/pci/pci.h | 6 +
hw/net/igb.c | 18 +
hw/net/igb_core.c | 144 ++-
hw/net/igb_migration.c | 1194 +++++++++++++++++++++++++
hw/net/igbvf.c | 29 +-
hw/pci/pci.c | 6 +-
hw/net/meson.build | 2 +-
hw/net/trace-events | 20 +
15 files changed, 1939 insertions(+), 26 deletions(-)
create mode 100644 docs/system/devices/igb-migration.rst
create mode 100644 hw/net/igb_migration.h
create mode 100644 hw/net/igb_migration.c
--
2.55.0
^ permalink raw reply [flat|nested] 26+ messages in thread
* [RFC PATCH 01/11] pci: Add PCI_BASE_ADDRESS_MEM_ALWAYS_ON BAR flag
2026-07-27 5:39 [RFC PATCH 00/11] igb: Add experimental VF live migration support Cédric Le Goater
@ 2026-07-27 5:39 ` Cédric Le Goater
2026-07-27 5:39 ` [RFC PATCH 02/11] igb: Add x-vf-migration property and vendor-specific capability for IGBVF Cédric Le Goater
` (11 subsequent siblings)
12 siblings, 0 replies; 26+ messages in thread
From: Cédric Le Goater @ 2026-07-27 5:39 UTC (permalink / raw)
To: qemu-devel
Cc: Akihiko Odaki, Sriram Yagnaraman, Jason Wang, Alex Williamson,
Michael S . Tsirkin, Peter Xu, Avihai Horon,
Cédric Le Goater
When VFIO opens a VF, it issues a Function Level Reset which clears
PCI_COMMAND_MEMORY. This unmaps all BARs, including the migration
BAR.
Add a QEMU-internal BAR type flag that keeps a memory BAR mapped
regardless of PCI_COMMAND_MEMORY. This is needed for host-only
control regions (e.g. migration BARs) that are accessed by a VFIO
variant driver, not by the guest.
Assisted-by: Claude
Signed-off-by: Cédric Le Goater <clg@redhat.com>
---
include/hw/pci/pci.h | 6 ++++++
hw/pci/pci.c | 6 ++++--
2 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h
index f2448e941a0b..923c8f0e15bc 100644
--- a/include/hw/pci/pci.h
+++ b/include/hw/pci/pci.h
@@ -164,6 +164,12 @@ typedef struct PCIIORegion {
MemoryRegion *address_space;
} PCIIORegion;
+/*
+ * QEMU-internal: keep this BAR mapped regardless of PCI_COMMAND_MEMORY.
+ * Stripped before writing to config space.
+ */
+#define PCI_BASE_ADDRESS_MEM_ALWAYS_ON 0x10
+
#define PCI_ROM_SLOT 6
#define PCI_NUM_REGIONS 7
diff --git a/hw/pci/pci.c b/hw/pci/pci.c
index d3191609e283..1394132c1472 100644
--- a/hw/pci/pci.c
+++ b/hw/pci/pci.c
@@ -1544,7 +1544,8 @@ void pci_register_bar(PCIDevice *pci_dev, int region_num,
}
addr = pci_bar(pci_dev, region_num);
- pci_set_long(pci_dev->config + addr, type);
+ pci_set_long(pci_dev->config + addr,
+ type & ~PCI_BASE_ADDRESS_MEM_ALWAYS_ON);
if (!(r->type & PCI_BASE_ADDRESS_SPACE_IO) &&
r->type & PCI_BASE_ADDRESS_MEM_TYPE_64) {
@@ -1686,7 +1687,8 @@ pcibus_t pci_bar_address(PCIDevice *d,
return new_addr;
}
- if (!(cmd & PCI_COMMAND_MEMORY)) {
+ if (!(cmd & PCI_COMMAND_MEMORY) &&
+ !(type & PCI_BASE_ADDRESS_MEM_ALWAYS_ON)) {
return PCI_BAR_UNMAPPED;
}
new_addr = pci_config_get_bar_addr(d, reg, type, size);
--
2.55.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [RFC PATCH 02/11] igb: Add x-vf-migration property and vendor-specific capability for IGBVF
2026-07-27 5:39 [RFC PATCH 00/11] igb: Add experimental VF live migration support Cédric Le Goater
2026-07-27 5:39 ` [RFC PATCH 01/11] pci: Add PCI_BASE_ADDRESS_MEM_ALWAYS_ON BAR flag Cédric Le Goater
@ 2026-07-27 5:39 ` Cédric Le Goater
2026-07-27 7:13 ` Akihiko Odaki
2026-07-27 5:39 ` [RFC PATCH 03/11] igb: Add migration BAR with state machine Cédric Le Goater
` (10 subsequent siblings)
12 siblings, 1 reply; 26+ messages in thread
From: Cédric Le Goater @ 2026-07-27 5:39 UTC (permalink / raw)
To: qemu-devel
Cc: Akihiko Odaki, Sriram Yagnaraman, Jason Wang, Alex Williamson,
Michael S . Tsirkin, Peter Xu, Avihai Horon,
Cédric Le Goater
Add a PF-level "x-vf-migration" boolean property (default off) that,
when enabled, causes each emulated VF to advertise a vendor-specific
PCI capability in its config space. The variant driver igb-vfio-pci
probes for this capability at bind time and uses its presence to
enable migration support.
The capability is a 16-byte vendor-specific capability (PCI_CAP_ID_VNDR)
containing a magic value ("MIGB"), the BAR index where the migration
region will be mapped, and feature flags indicating which migration
features are supported.
Assisted-by: Claude
Suggested-by: Alex Williamson <alex@shazbot.org>
Signed-off-by: Cédric Le Goater <clg@redhat.com>
---
MAINTAINERS | 6 +++
docs/system/device-emulation.rst | 1 +
docs/system/devices/igb-migration.rst | 20 ++++++++
docs/system/devices/igb.rst | 6 +++
hw/net/igb_migration.h | 43 ++++++++++++++++
hw/net/igb.c | 12 +++++
hw/net/igb_migration.c | 72 +++++++++++++++++++++++++++
hw/net/igbvf.c | 11 ++++
hw/net/meson.build | 2 +-
hw/net/trace-events | 1 +
10 files changed, 173 insertions(+), 1 deletion(-)
create mode 100644 docs/system/devices/igb-migration.rst
create mode 100644 hw/net/igb_migration.h
create mode 100644 hw/net/igb_migration.c
diff --git a/MAINTAINERS b/MAINTAINERS
index a28935c89866..6d29b905c47c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2765,6 +2765,12 @@ F: tests/functional/x86_64/test_netdev_ethtool.py
F: tests/qtest/igb-test.c
F: tests/qtest/libqos/igb.c
+igb VF migration
+M: Cédric Le Goater <clg@redhat.com>
+S: Maintained
+F: hw/net/igb_migration.*
+F: docs/system/devices/igb-migration.rst
+
eepro100
M: Stefan Weil <sw@weilnetz.de>
S: Maintained
diff --git a/docs/system/device-emulation.rst b/docs/system/device-emulation.rst
index 40054bb7dfcc..75f423b795cd 100644
--- a/docs/system/device-emulation.rst
+++ b/docs/system/device-emulation.rst
@@ -90,6 +90,7 @@ Emulated Devices
devices/cxl.rst
devices/emmc.rst
devices/igb.rst
+ devices/igb-migration.rst
devices/ivshmem-flat.rst
devices/ivshmem.rst
devices/keyboard.rst
diff --git a/docs/system/devices/igb-migration.rst b/docs/system/devices/igb-migration.rst
new file mode 100644
index 000000000000..017b47c4544e
--- /dev/null
+++ b/docs/system/devices/igb-migration.rst
@@ -0,0 +1,20 @@
+.. SPDX-License-Identifier: GPL-2.0-or-later
+.. _igb-migration:
+
+igb VF Migration
+----------------
+
+The igb device supports an experimental VF migration interface that allows
+the ``igb-vfio-pci`` variant driver to migrate VF state during live
+migration. This is enabled with the ``x-vf-migration`` property::
+
+ -device igb,x-vf-migration=on,...
+
+When enabled, each emulated VF advertises a vendor-specific PCI capability
+(cap id 0x09) with a magic signature (``0x4D494742`` / "MIGB") that the
+variant driver probes at bind time. The capability contains an interface
+version number, the BAR index hosting the migration register region, and
+feature flags indicating which migration features are supported.
+
+This feature is experimental and the ``x-`` prefix indicates the interface
+may change.
diff --git a/docs/system/devices/igb.rst b/docs/system/devices/igb.rst
index 50f625fd77e4..00271dbc92c3 100644
--- a/docs/system/devices/igb.rst
+++ b/docs/system/devices/igb.rst
@@ -64,6 +64,12 @@ command:
pyvenv/bin/meson test --suite thorough func-x86_64-netdev_ethtool
+VF Migration (experimental)
+===========================
+
+See :ref:`igb-migration` for details on the experimental VF live migration
+interface.
+
References
==========
diff --git a/hw/net/igb_migration.h b/hw/net/igb_migration.h
new file mode 100644
index 000000000000..e79892436b21
--- /dev/null
+++ b/hw/net/igb_migration.h
@@ -0,0 +1,43 @@
+/*
+ * QEMU Intel 82576 SR/IOV VF Migration Support
+ *
+ * Copyright (c) 2026 Red Hat, Inc.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#ifndef IGB_MIGRATION_H
+#define IGB_MIGRATION_H
+
+#include "hw/pci/pci_device.h"
+
+/* Migration BAR definitions */
+#define IGB_MIG_BAR_IDX (2)
+#define IGB_MIG_BAR_SIZE (64 * 1024)
+
+/*
+ * Vendor-specific PCI capability for migration discovery.
+ *
+ * The igb-vfio-pci variant driver probes for this at bind time.
+ * If present, the driver knows the emulated VF supports migration.
+ */
+#define IGB_MIG_CAP_MAGIC 0x4D494742 /* "MIGB" */
+#define IGB_MIG_CAP_VERSION 1
+
+#define IGB_MIG_CAP_F_STATE (1u << 0) /* device state serialization */
+#define IGB_MIG_CAP_F_DIRTY (1u << 1) /* dirty page tracking */
+
+#define IGB_MIG_CAP_SIZE 16
+#define IGB_MIG_CAP_OFF_MAGIC 4 /* offset within cap for magic field */
+#define IGB_MIG_CAP_OFF_BARID 8 /* offset within cap for BAR id */
+#define IGB_MIG_CAP_OFF_FLAGS 12 /* offset within cap for feature flags */
+
+typedef struct IgbVfMigState {
+ bool migration_cap;
+ MemoryRegion mig_bar;
+} IgbVfMigState;
+
+void igb_pf_init_migration_bar(PCIDevice *dev);
+bool igbvf_add_migration_cap(PCIDevice *dev, Error **errp);
+
+#endif
diff --git a/hw/net/igb.c b/hw/net/igb.c
index c076807e7110..b43235996db2 100644
--- a/hw/net/igb.c
+++ b/hw/net/igb.c
@@ -57,6 +57,7 @@
#include "igb_common.h"
#include "igb_core.h"
+#include "igb_migration.h"
#include "trace.h"
#include "qapi/error.h"
@@ -69,6 +70,7 @@ struct IGBState {
PCIDevice parent_obj;
NICState *nic;
NICConf conf;
+ bool vf_migration;
MemoryRegion mmio;
MemoryRegion flash;
@@ -459,6 +461,15 @@ static void igb_pci_realize(PCIDevice *pci_dev, Error **errp)
pcie_sriov_pf_init_vf_bar(pci_dev, IGBVF_MSIX_BAR_IDX,
PCI_BASE_ADDRESS_MEM_TYPE_64 | PCI_BASE_ADDRESS_MEM_PREFETCH,
IGBVF_MSIX_SIZE);
+ /*
+ * When VF migration support is enabled, register an additional VF
+ * BAR for the migration register region. The variant driver
+ * discovers this via a vendor-specific PCI capability that points
+ * to this BAR.
+ */
+ if (s->vf_migration) {
+ igb_pf_init_migration_bar(pci_dev);
+ }
igb_init_net_peer(s, pci_dev, macaddr);
@@ -597,6 +608,7 @@ static const VMStateDescription igb_vmstate = {
static const Property igb_properties[] = {
DEFINE_NIC_PROPERTIES(IGBState, conf),
DEFINE_PROP_BOOL("x-pcie-flr-init", IGBState, has_flr, true),
+ DEFINE_PROP_BOOL("x-vf-migration", IGBState, vf_migration, false),
};
static void igb_class_init(ObjectClass *class, const void *data)
diff --git a/hw/net/igb_migration.c b/hw/net/igb_migration.c
new file mode 100644
index 000000000000..794f1217d3e5
--- /dev/null
+++ b/hw/net/igb_migration.c
@@ -0,0 +1,72 @@
+/*
+ * QEMU Intel 82576 SR/IOV VF Migration Support
+ *
+ * Copyright (c) 2026 Red Hat, Inc.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#include "qemu/osdep.h"
+#include "hw/pci/pci_device.h"
+#include "hw/pci/pcie.h"
+#include "igb_common.h"
+#include "igb_migration.h"
+#include "trace.h"
+
+
+
+/*
+ * 32-bit prefetchable BAR. A 64-bit BAR2 would consume BAR2+BAR3, but
+ * BAR3 is already used for MSI-X (IGBVF_MSIX_BAR_IDX = 3).
+ *
+ * BAR Index Type Size Purpose
+ * BAR0 0+1 64-bit prefetchable 16 KB MMIO registers
+ * BAR2 2 32-bit prefetchable 64 KB Migration
+ * BAR3 3+4 64-bit prefetchable 16 KB MSI-X table + PBA
+ * BAR5 5 - - Unused
+ */
+void igb_pf_init_migration_bar(PCIDevice *dev)
+{
+ pcie_sriov_pf_init_vf_bar(dev, IGB_MIG_BAR_IDX,
+ PCI_BASE_ADDRESS_MEM_PREFETCH,
+ IGB_MIG_BAR_SIZE);
+}
+
+/*
+ * Add vendor-specific PCI capability that the variant driver probes for.
+ *
+ * Layout (16 bytes):
+ * [0] cap_id (PCI_CAP_ID_VNDR = 0x09)
+ * [1] next_cap
+ * [2] cap_len (16)
+ * [3] version (IGB_MIG_CAP_VERSION)
+ * [4-7] magic (IGB_MIG_CAP_MAGIC, little-endian)
+ * [8-11] bar_id (IGB_MIG_BAR_IDX, little-endian)
+ * [12-15] flags (feature flags, little-endian)
+ */
+bool igbvf_add_migration_cap(PCIDevice *dev, Error **errp)
+{
+ int offset;
+
+ offset = pci_add_capability(dev, PCI_CAP_ID_VNDR, 0,
+ IGB_MIG_CAP_SIZE, errp);
+ if (offset < 0) {
+ return false;
+ }
+
+ /* Length and version in the standard cap flags word */
+ pci_set_byte(dev->config + offset + PCI_CAP_FLAGS,
+ IGB_MIG_CAP_SIZE);
+ pci_set_byte(dev->config + offset + PCI_CAP_FLAGS + 1,
+ IGB_MIG_CAP_VERSION);
+
+ pci_set_long(dev->config + offset + IGB_MIG_CAP_OFF_MAGIC,
+ IGB_MIG_CAP_MAGIC);
+ pci_set_long(dev->config + offset + IGB_MIG_CAP_OFF_BARID,
+ IGB_MIG_BAR_IDX);
+ pci_set_long(dev->config + offset + IGB_MIG_CAP_OFF_FLAGS,
+ IGB_MIG_CAP_F_STATE);
+
+ trace_igbvf_mig_cap_add(pcie_sriov_vf_number(dev), offset);
+ return true;
+}
diff --git a/hw/net/igbvf.c b/hw/net/igbvf.c
index 9a165c7063ee..94c9739cd58c 100644
--- a/hw/net/igbvf.c
+++ b/hw/net/igbvf.c
@@ -47,6 +47,7 @@
#include "net/net.h"
#include "igb_common.h"
#include "igb_core.h"
+#include "igb_migration.h"
#include "trace.h"
#include "qapi/error.h"
@@ -57,6 +58,8 @@ struct IgbVfState {
MemoryRegion mmio;
MemoryRegion msix;
+
+ IgbVfMigState mig;
};
static hwaddr vf_to_pf_addr(hwaddr addr, uint16_t vfn, bool write)
@@ -272,6 +275,14 @@ static void igbvf_pci_realize(PCIDevice *dev, Error **errp)
hw_error("Failed to initialize PCIe capability");
}
+ s->mig.migration_cap = object_property_get_bool(OBJECT(pcie_sriov_get_pf(dev)),
+ "x-vf-migration", &error_abort);
+ if (s->mig.migration_cap) {
+ if (!igbvf_add_migration_cap(dev, errp)) {
+ return;
+ }
+ }
+
if (object_property_get_bool(OBJECT(pcie_sriov_get_pf(dev)),
"x-pcie-flr-init", &error_abort)) {
pcie_cap_flr_init(dev);
diff --git a/hw/net/meson.build b/hw/net/meson.build
index 84f142df222a..bb4b449b25ba 100644
--- a/hw/net/meson.build
+++ b/hw/net/meson.build
@@ -11,7 +11,7 @@ system_ss.add(when: 'CONFIG_E1000_PCI', if_true: files('e1000.c', 'e1000x_common
system_ss.add(when: 'CONFIG_E1000E_PCI_EXPRESS', if_true: files('net_tx_pkt.c', 'net_rx_pkt.c'))
system_ss.add(when: 'CONFIG_E1000E_PCI_EXPRESS', if_true: files('e1000e.c', 'e1000e_core.c', 'e1000x_common.c'))
system_ss.add(when: 'CONFIG_IGB_PCI_EXPRESS', if_true: files('net_tx_pkt.c', 'net_rx_pkt.c'))
-system_ss.add(when: 'CONFIG_IGB_PCI_EXPRESS', if_true: files('igb.c', 'igbvf.c', 'igb_core.c'))
+system_ss.add(when: 'CONFIG_IGB_PCI_EXPRESS', if_true: files('igb.c', 'igbvf.c', 'igb_core.c', 'igb_migration.c'))
system_ss.add(when: 'CONFIG_RTL8139_PCI', if_true: files('rtl8139.c'))
system_ss.add(when: 'CONFIG_TULIP', if_true: files('tulip.c'))
system_ss.add(when: 'CONFIG_VMXNET3_PCI', if_true: files('net_tx_pkt.c', 'net_rx_pkt.c'))
diff --git a/hw/net/trace-events b/hw/net/trace-events
index 001a20b0e2ac..06d8848023e5 100644
--- a/hw/net/trace-events
+++ b/hw/net/trace-events
@@ -294,6 +294,7 @@ igb_wrn_rx_desc_modes_not_supp(int desc_type) "Not supported descriptor type: %d
# igbvf.c
igbvf_wrn_io_addr_unknown(uint64_t addr) "IO unknown register 0x%"PRIx64
+igbvf_mig_cap_add(uint16_t vfn, int offset) "VF%u: added migration vendor cap at config offset 0x%x"
# spapr_llan.c
spapr_vlan_get_rx_bd_from_pool_found(int pool, int32_t count, uint32_t rx_bufs) "pool=%d count=%"PRId32" rxbufs=%"PRIu32
--
2.55.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [RFC PATCH 03/11] igb: Add migration BAR with state machine
2026-07-27 5:39 [RFC PATCH 00/11] igb: Add experimental VF live migration support Cédric Le Goater
2026-07-27 5:39 ` [RFC PATCH 01/11] pci: Add PCI_BASE_ADDRESS_MEM_ALWAYS_ON BAR flag Cédric Le Goater
2026-07-27 5:39 ` [RFC PATCH 02/11] igb: Add x-vf-migration property and vendor-specific capability for IGBVF Cédric Le Goater
@ 2026-07-27 5:39 ` Cédric Le Goater
2026-07-27 7:16 ` Akihiko Odaki
2026-07-27 5:39 ` [RFC PATCH 04/11] igb: Add VF state serialization for live migration Cédric Le Goater
` (9 subsequent siblings)
12 siblings, 1 reply; 26+ messages in thread
From: Cédric Le Goater @ 2026-07-27 5:39 UTC (permalink / raw)
To: qemu-devel
Cc: Akihiko Odaki, Sriram Yagnaraman, Jason Wang, Alex Williamson,
Michael S . Tsirkin, Peter Xu, Avihai Horon,
Cédric Le Goater
When x-vf-migration=on, register a 64KB migration BAR (BAR2) on each
emulated VF. This BAR implements a VFIO-like migration state machine
that the igb-vfio-pci variant driver uses to serialize/deserialize VF
device state during live migration.
The migration register region is laid out as:
0x000 DEVICE_STATE (RW) - migration state machine control
0x004 MIG_STATUS (RO) - status flags (DATA_AVAIL, ERROR,
QUIESCED) + error code in [15:8]
0x008 MIG_CAPS (RO) - advertised capabilities
0x00C MIG_VERSION (RO) - interface version
0x010 DATA_SIZE (RW) - max state size at reset, actual after save
0x014 DATA_XFER (WO) - trigger DMA save or DMA load
0x018 DATA_BUF_ADDR_LO (WO) - low 32 bits of state DMA buffer
0x01C DATA_BUF_ADDR_HI (WO) - high 32 bits of state DMA buffer
State data is transferred via a driver-provided DMA buffer. The driver
writes its PF DMA address to DATA_BUF_ADDR_LO/HI and triggers the
transfer with DATA_XFER. The device DMA-writes the serialized state on
save and DMA-reads it on restore. DMA is performed through the PF
device (pcie_sriov_get_pf) because VFIO owns the VF's IOMMU domain.
VF state serialization is added in the next patch.
Assisted-by: Claude
Suggested-by: Alex Williamson <alex@shazbot.org>
Signed-off-by: Cédric Le Goater <clg@redhat.com>
---
docs/system/devices/igb-migration.rst | 43 ++++
hw/net/igb_common.h | 11 +
hw/net/igb_migration.h | 52 +++++
hw/net/igb_migration.c | 293 ++++++++++++++++++++++++++
hw/net/igbvf.c | 18 +-
hw/net/trace-events | 8 +
6 files changed, 416 insertions(+), 9 deletions(-)
diff --git a/docs/system/devices/igb-migration.rst b/docs/system/devices/igb-migration.rst
index 017b47c4544e..04c44ef072af 100644
--- a/docs/system/devices/igb-migration.rst
+++ b/docs/system/devices/igb-migration.rst
@@ -18,3 +18,46 @@ feature flags indicating which migration features are supported.
This feature is experimental and the ``x-`` prefix indicates the interface
may change.
+
+Migration BAR layout
+~~~~~~~~~~~~~~~~~~~~
+
+The migration BAR (BAR2, 64 KB) implements a VFIO-like state machine with
+the following register layout::
+
+ Offset Name Access Description
+ 0x000 DEVICE_STATE RW Migration state (RUNNING=2, STOP=1,
+ STOP_COPY=3, RESUMING=4, PRE_COPY=5)
+ 0x004 STATUS RO Flags[2:0]: DATA_AVAIL, ERROR, QUIESCED
+ Error code[15:8] (when ERROR is set)
+ 0x008 CAPS RO F_STATE, F_DIRTY, max_ranges[11:8],
+ pgsizes[31:12]
+ 0x00C VERSION RO Interface version (1)
+ 0x010 DATA_SIZE RW Max state size at reset, actual after save
+ 0x014 DATA_XFER WO Trigger DMA save or DMA load
+ 0x018 DATA_BUF_ADDR_LO WO Low 32 bits of state DMA buffer address
+ 0x01C DATA_BUF_ADDR_HI WO High 32 bits of state DMA buffer address
+
+State transitions follow the VFIO migration state machine: the driver
+writes to ``DEVICE_STATE`` to move between states and reads ``STATUS``
+to check for completion.
+
+State data is transferred via a driver-provided DMA buffer. The driver
+writes its PF DMA address to ``DATA_BUF_ADDR_LO/HI`` and triggers the
+transfer with ``DATA_XFER``. The device DMA-writes the serialized state
+on save and DMA-reads it on restore. DMA is performed through the PF
+device because VFIO owns the VF's IOMMU domain.
+
+The state blob is a versioned sequence of register (offset, value)
+pairs.
+
+When ``STATUS`` has the ``ERROR`` bit set, bits [15:8] contain an error
+code identifying the failure::
+
+ 0 (none) No error
+ 1 BAD_MAGIC State blob magic mismatch
+ 2 BAD_VERSION State blob version mismatch
+ 3 BAD_SIZE State blob too large or empty
+ 4 BAD_VFN VF number mismatch (source != destination)
+ 5 DMA_FAILED DMA transfer to/from state buffer failed
+ 6 NO_BUFFER DATA_XFER without buffer address set
diff --git a/hw/net/igb_common.h b/hw/net/igb_common.h
index b316a5bcfa5c..01816e002c23 100644
--- a/hw/net/igb_common.h
+++ b/hw/net/igb_common.h
@@ -27,6 +27,7 @@
#define HW_NET_IGB_COMMON_H
#include "igb_regs.h"
+#include "igb_migration.h"
#define TYPE_IGBVF "igbvf"
@@ -36,6 +37,16 @@
#define IGBVF_MMIO_SIZE (16 * 1024)
#define IGBVF_MSIX_SIZE (16 * 1024)
+struct IgbVfState {
+ PCIDevice parent_obj;
+ uint16_t vfn;
+
+ MemoryRegion mmio;
+ MemoryRegion msix;
+
+ IgbVfMigState mig;
+};
+
#define defreg(x) x = (E1000_##x >> 2)
#define defreg_indexed(x, i) x##i = (E1000_##x(i) >> 2)
#define defreg_indexeda(x, i) x##i##_A = (E1000_##x##_A(i) >> 2)
diff --git a/hw/net/igb_migration.h b/hw/net/igb_migration.h
index e79892436b21..739a189810b0 100644
--- a/hw/net/igb_migration.h
+++ b/hw/net/igb_migration.h
@@ -32,12 +32,64 @@
#define IGB_MIG_CAP_OFF_BARID 8 /* offset within cap for BAR id */
#define IGB_MIG_CAP_OFF_FLAGS 12 /* offset within cap for feature flags */
+/*
+ * Maximum serialized VF state size, sized to hold all per-VF
+ * registers plus TX context descriptors with room to spare.
+ */
+#define IGB_VF_STATE_MAX_SIZE 4096
+
+/*
+ * Migration BAR register offsets.
+ */
+#define IGB_MIG_DEVICE_STATE 0x000
+#define IGB_MIG_STATUS 0x004
+#define IGB_MIG_CAPS 0x008
+#define IGB_MIG_VERSION 0x00C
+#define IGB_MIG_DATA_SIZE 0x010
+#define IGB_MIG_DATA_XFER 0x014
+#define IGB_MIG_DATA_BUF_ADDR_LO 0x018
+#define IGB_MIG_DATA_BUF_ADDR_HI 0x01C
+
+/* DEVICE_STATE values - mirrors VFIO migration states */
+#define IGB_MIG_STATE_ERROR 0
+#define IGB_MIG_STATE_STOP 1
+#define IGB_MIG_STATE_RUNNING 2
+#define IGB_MIG_STATE_STOP_COPY 3
+#define IGB_MIG_STATE_RESUMING 4
+#define IGB_MIG_STATE_PRE_COPY 5
+
+/* MIG_STATUS bits */
+#define IGB_MIG_STATUS_DATA_AVAIL (1u << 0)
+#define IGB_MIG_STATUS_ERROR (1u << 1)
+
+/* MIG_STATUS error codes in bits [15:8], valid when ERROR bit is set */
+#define IGB_MIG_STATUS_ERR_SHIFT 8
+#define IGB_MIG_STATUS_ERR_MASK (0xffu << IGB_MIG_STATUS_ERR_SHIFT)
+#define IGB_MIG_STATUS_ERR(code) (IGB_MIG_STATUS_ERROR | \
+ ((uint32_t)(code) << IGB_MIG_STATUS_ERR_SHIFT))
+
+#define IGB_MIG_ERR_BAD_MAGIC 1
+#define IGB_MIG_ERR_BAD_VERSION 2
+#define IGB_MIG_ERR_BAD_SIZE 3
+#define IGB_MIG_ERR_BAD_VFN 4
+#define IGB_MIG_ERR_DMA_FAILED 5
+#define IGB_MIG_ERR_NO_BUFFER 6
+
typedef struct IgbVfMigState {
bool migration_cap;
MemoryRegion mig_bar;
+
+ uint32_t mig_state;
+ uint8_t mig_error;
+ uint8_t mig_data[IGB_VF_STATE_MAX_SIZE];
+ uint32_t mig_data_size;
+ uint64_t mig_data_buf_addr;
} IgbVfMigState;
+typedef struct IgbVfState IgbVfState;
void igb_pf_init_migration_bar(PCIDevice *dev);
bool igbvf_add_migration_cap(PCIDevice *dev, Error **errp);
+void igbvf_mig_bar_init(IgbVfState *s);
+void igbvf_mig_state_reset(IgbVfState *s);
#endif
diff --git a/hw/net/igb_migration.c b/hw/net/igb_migration.c
index 794f1217d3e5..a0d044815fd9 100644
--- a/hw/net/igb_migration.c
+++ b/hw/net/igb_migration.c
@@ -7,9 +7,13 @@
*/
#include "qemu/osdep.h"
+#include "qemu/log.h"
#include "hw/pci/pci_device.h"
#include "hw/pci/pcie.h"
+#include "net/eth.h"
+#include "net/net.h"
#include "igb_common.h"
+#include "igb_core.h"
#include "igb_migration.h"
#include "trace.h"
@@ -70,3 +74,292 @@ bool igbvf_add_migration_cap(PCIDevice *dev, Error **errp)
trace_igbvf_mig_cap_add(pcie_sriov_vf_number(dev), offset);
return true;
}
+
+/*
+ * =====================================================================
+ * Per-VF state serialization / deserialization
+ * =====================================================================
+ */
+
+static int igb_core_vf_save_state(IgbVfState *s,
+ void *buf, size_t buf_size)
+{
+ int size = 0;
+
+ trace_igbvf_mig_save_state(s->vfn, size);
+ return size;
+}
+
+static int igb_core_vf_max_data_size(IgbVfState *s)
+{
+ int size = igb_core_vf_save_state(s, NULL, 0);
+
+ g_assert(size > 0 && size <= IGB_VF_STATE_MAX_SIZE);
+ return size;
+}
+
+static int igb_core_vf_load_state(IgbVfState *s,
+ const void *buf, size_t size)
+{
+ trace_igbvf_mig_load_state(s->vfn, (uint32_t)size);
+ return 0;
+}
+
+static int igbvf_mig_load(IgbVfState *s, const void *buf, size_t size)
+{
+ int ret;
+
+ ret = igb_core_vf_load_state(s, buf, size);
+ if (ret < 0) {
+ return ret;
+ }
+
+ return 0;
+}
+
+/* ================================================================
+ * Migration BAR register read/write handlers
+ * ================================================================ */
+
+static bool igbvf_mig_set_state(IgbVfState *s, uint32_t new_state)
+{
+ IgbVfMigState *ms = &s->mig;
+ uint32_t old = ms->mig_state;
+ int ret;
+
+ switch (new_state) {
+ case IGB_MIG_STATE_STOP:
+ if (old != IGB_MIG_STATE_RUNNING &&
+ old != IGB_MIG_STATE_STOP_COPY &&
+ old != IGB_MIG_STATE_RESUMING &&
+ old != IGB_MIG_STATE_ERROR) {
+ return false;
+ }
+ /* Restore DATA_SIZE to max, same as at reset */
+ ms->mig_data_size = igb_core_vf_max_data_size(s);
+ break;
+
+ case IGB_MIG_STATE_RUNNING:
+ if (old != IGB_MIG_STATE_STOP) {
+ return false;
+ }
+ break;
+
+ case IGB_MIG_STATE_STOP_COPY:
+ if (old != IGB_MIG_STATE_STOP) {
+ return false;
+ }
+ ret = igb_core_vf_save_state(s, ms->mig_data, sizeof(ms->mig_data));
+ if (ret < 0) {
+ ms->mig_error = -ret;
+ ms->mig_state = IGB_MIG_STATE_ERROR;
+ return false;
+ }
+ ms->mig_data_size = ret;
+ break;
+
+ case IGB_MIG_STATE_RESUMING:
+ if (old != IGB_MIG_STATE_STOP) {
+ return false;
+ }
+ memset(ms->mig_data, 0, sizeof(ms->mig_data));
+ ms->mig_data_size = 0;
+ break;
+
+ default:
+ trace_igbvf_mig_set_state_err(s->vfn, old, new_state);
+ return false;
+ }
+
+ ms->mig_state = new_state;
+ trace_igbvf_mig_set_state(s->vfn, old, new_state);
+ return true;
+}
+
+static uint32_t igbvf_mig_get_status(IgbVfState *s)
+{
+ IgbVfMigState *ms = &s->mig;
+ uint32_t status = 0;
+
+ if (ms->mig_state == IGB_MIG_STATE_ERROR) {
+ status |= IGB_MIG_STATUS_ERR(ms->mig_error);
+ }
+ if (ms->mig_state == IGB_MIG_STATE_STOP_COPY && ms->mig_data_size > 0) {
+ status |= IGB_MIG_STATUS_DATA_AVAIL;
+ }
+
+ return status;
+}
+
+static void igbvf_mig_data_xfer(IgbVfState *s, uint32_t val)
+{
+ IgbVfMigState *ms = &s->mig;
+ MemTxResult r;
+ int ret;
+
+ if (!ms->mig_data_buf_addr) {
+ ms->mig_error = IGB_MIG_ERR_NO_BUFFER;
+ ms->mig_state = IGB_MIG_STATE_ERROR;
+ return;
+ }
+
+ switch (ms->mig_state) {
+ case IGB_MIG_STATE_STOP_COPY:
+ /* Save: DMA-write serialized state to driver buffer */
+ r = pci_dma_write(pcie_sriov_get_pf(PCI_DEVICE(s)),
+ ms->mig_data_buf_addr,
+ ms->mig_data, ms->mig_data_size);
+ if (r != MEMTX_OK) {
+ qemu_log_mask(LOG_GUEST_ERROR,
+ "igbvf: VF%u state write failed at 0x%" PRIx64 "\n",
+ s->vfn, ms->mig_data_buf_addr);
+ ms->mig_error = IGB_MIG_ERR_DMA_FAILED;
+ ms->mig_state = IGB_MIG_STATE_ERROR;
+ }
+ break;
+
+ case IGB_MIG_STATE_RESUMING:
+ /* Restore: DMA-read state from driver buffer and deserialize */
+ if (ms->mig_data_size == 0 ||
+ ms->mig_data_size > sizeof(ms->mig_data)) {
+ ms->mig_error = IGB_MIG_ERR_BAD_SIZE;
+ ms->mig_state = IGB_MIG_STATE_ERROR;
+ break;
+ }
+
+ r = pci_dma_read(pcie_sriov_get_pf(PCI_DEVICE(s)),
+ ms->mig_data_buf_addr,
+ ms->mig_data, ms->mig_data_size);
+ if (r != MEMTX_OK) {
+ qemu_log_mask(LOG_GUEST_ERROR,
+ "igbvf: VF%u state read failed at 0x%" PRIx64 "\n",
+ s->vfn, ms->mig_data_buf_addr);
+ ms->mig_error = IGB_MIG_ERR_DMA_FAILED;
+ ms->mig_state = IGB_MIG_STATE_ERROR;
+ break;
+ }
+
+ ret = igbvf_mig_load(s, ms->mig_data, ms->mig_data_size);
+ if (ret < 0) {
+ ms->mig_error = -ret;
+ ms->mig_state = IGB_MIG_STATE_ERROR;
+ }
+ break;
+
+ default:
+ break;
+ }
+}
+
+static uint64_t igbvf_mig_read(void *opaque, hwaddr addr, unsigned size)
+{
+ IgbVfState *s = opaque;
+ IgbVfMigState *ms = &s->mig;
+ uint64_t val = 0;
+
+ switch (addr) {
+ case IGB_MIG_DEVICE_STATE:
+ val = ms->mig_state;
+ break;
+ case IGB_MIG_STATUS:
+ val = igbvf_mig_get_status(s);
+ break;
+ case IGB_MIG_CAPS:
+ val = IGB_MIG_CAP_F_STATE;
+ break;
+ case IGB_MIG_VERSION:
+ val = IGB_MIG_CAP_VERSION;
+ break;
+ case IGB_MIG_DATA_SIZE:
+ val = ms->mig_data_size;
+ break;
+ default:
+ qemu_log_mask(LOG_GUEST_ERROR,
+ "igbvf: VF%u bad migration BAR read at 0x%"
+ HWADDR_PRIx "\n", s->vfn, addr);
+ break;
+ }
+
+ trace_igbvf_mig_bar_read(s->vfn, addr, val);
+
+ return val;
+}
+
+static void igbvf_mig_write(void *opaque, hwaddr addr, uint64_t val,
+ unsigned size)
+{
+ IgbVfState *s = opaque;
+ IgbVfMigState *ms = &s->mig;
+
+ trace_igbvf_mig_bar_write(s->vfn, addr, val);
+
+ switch (addr) {
+ case IGB_MIG_DEVICE_STATE:
+ igbvf_mig_set_state(s, (uint32_t)val);
+ break;
+ case IGB_MIG_DATA_SIZE:
+ if (val <= sizeof(ms->mig_data)) {
+ ms->mig_data_size = (uint32_t)val;
+ } else {
+ qemu_log_mask(LOG_GUEST_ERROR,
+ "igbvf: VF%u DATA_SIZE %" PRIu64 " exceeds max %zu\n",
+ s->vfn, val, sizeof(ms->mig_data));
+ }
+ break;
+ case IGB_MIG_DATA_XFER:
+ igbvf_mig_data_xfer(s, (uint32_t)val);
+ break;
+ case IGB_MIG_DATA_BUF_ADDR_LO:
+ ms->mig_data_buf_addr =
+ deposit64(ms->mig_data_buf_addr, 0, 32, val);
+ break;
+ case IGB_MIG_DATA_BUF_ADDR_HI:
+ ms->mig_data_buf_addr =
+ deposit64(ms->mig_data_buf_addr, 32, 32, val);
+ break;
+ default:
+ qemu_log_mask(LOG_GUEST_ERROR,
+ "igbvf: VF%u bad migration BAR write at 0x%"
+ HWADDR_PRIx "\n", s->vfn, addr);
+ break;
+ }
+}
+
+static const MemoryRegionOps mig_bar_ops = {
+ .read = igbvf_mig_read,
+ .write = igbvf_mig_write,
+ .endianness = DEVICE_LITTLE_ENDIAN,
+ .impl = {
+ .min_access_size = 4,
+ .max_access_size = 4,
+ },
+};
+
+/*
+ * Use the QEM-internal PCI_BASE_ADDRESS_MEM_ALWAYS_ON BAR type flag
+ * to keep the memory BAR always mapped.
+ */
+void igbvf_mig_bar_init(IgbVfState *s)
+{
+ IgbVfMigState *ms = &s->mig;
+
+ memory_region_init_io(&ms->mig_bar, OBJECT(s), &mig_bar_ops, s,
+ "igbvf-mig", IGB_MIG_BAR_SIZE);
+ pci_register_bar(PCI_DEVICE(s), IGB_MIG_BAR_IDX,
+ PCI_BASE_ADDRESS_MEM_PREFETCH |
+ PCI_BASE_ADDRESS_MEM_ALWAYS_ON,
+ &ms->mig_bar);
+ trace_igbvf_mig_bar_init(s->vfn);
+}
+
+void igbvf_mig_state_reset(IgbVfState *s)
+{
+ IgbVfMigState *ms = &s->mig;
+
+ ms->mig_state = IGB_MIG_STATE_RUNNING;
+ ms->mig_error = 0;
+ ms->mig_data_size = igb_core_vf_max_data_size(s);
+ ms->mig_data_buf_addr = 0;
+ memset(ms->mig_data, 0, sizeof(ms->mig_data));
+ trace_igbvf_mig_reset(s->vfn);
+}
diff --git a/hw/net/igbvf.c b/hw/net/igbvf.c
index 94c9739cd58c..e9f9fc3369d8 100644
--- a/hw/net/igbvf.c
+++ b/hw/net/igbvf.c
@@ -53,15 +53,6 @@
OBJECT_DECLARE_SIMPLE_TYPE(IgbVfState, IGBVF)
-struct IgbVfState {
- PCIDevice parent_obj;
-
- MemoryRegion mmio;
- MemoryRegion msix;
-
- IgbVfMigState mig;
-};
-
static hwaddr vf_to_pf_addr(hwaddr addr, uint16_t vfn, bool write)
{
switch (addr) {
@@ -281,6 +272,10 @@ static void igbvf_pci_realize(PCIDevice *dev, Error **errp)
if (!igbvf_add_migration_cap(dev, errp)) {
return;
}
+
+ s->vfn = pcie_sriov_vf_number(dev);
+ igbvf_mig_bar_init(s);
+ igbvf_mig_state_reset(s);
}
if (object_property_get_bool(OBJECT(pcie_sriov_get_pf(dev)),
@@ -298,8 +293,13 @@ static void igbvf_pci_realize(PCIDevice *dev, Error **errp)
static void igbvf_qdev_reset_hold(Object *obj, ResetType type)
{
PCIDevice *vf = PCI_DEVICE(obj);
+ IgbVfState *s = IGBVF(obj);
igb_vf_reset(pcie_sriov_get_pf(vf), pcie_sriov_vf_number(vf));
+
+ if (s->mig.migration_cap) {
+ igbvf_mig_state_reset(s);
+ }
}
static void igbvf_pci_uninit(PCIDevice *dev)
diff --git a/hw/net/trace-events b/hw/net/trace-events
index 06d8848023e5..0b13a99b3f32 100644
--- a/hw/net/trace-events
+++ b/hw/net/trace-events
@@ -295,6 +295,14 @@ igb_wrn_rx_desc_modes_not_supp(int desc_type) "Not supported descriptor type: %d
# igbvf.c
igbvf_wrn_io_addr_unknown(uint64_t addr) "IO unknown register 0x%"PRIx64
igbvf_mig_cap_add(uint16_t vfn, int offset) "VF%u: added migration vendor cap at config offset 0x%x"
+igbvf_mig_bar_init(uint16_t vfn) "VF%u: migration BAR initialized"
+igbvf_mig_bar_read(uint16_t vfn, uint64_t addr, uint64_t val) "VF%u: BAR read addr=0x%"PRIx64" val=0x%"PRIx64
+igbvf_mig_bar_write(uint16_t vfn, uint64_t addr, uint64_t val) "VF%u: BAR write addr=0x%"PRIx64" val=0x%"PRIx64
+igbvf_mig_set_state(uint16_t vfn, uint32_t old_state, uint32_t new_state) "VF%u: state %u -> %u"
+igbvf_mig_set_state_err(uint16_t vfn, uint32_t old_state, uint32_t new_state) "VF%u: invalid transition %u -> %u"
+igbvf_mig_save_state(uint16_t vfn, uint32_t size) "VF%u: saved %u bytes of device state"
+igbvf_mig_load_state(uint16_t vfn, uint32_t size) "VF%u: loaded %u bytes of device state"
+igbvf_mig_reset(uint16_t vfn) "VF%u: migration state reset"
# spapr_llan.c
spapr_vlan_get_rx_bd_from_pool_found(int pool, int32_t count, uint32_t rx_bufs) "pool=%d count=%"PRId32" rxbufs=%"PRIu32
--
2.55.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [RFC PATCH 04/11] igb: Add VF state serialization for live migration
2026-07-27 5:39 [RFC PATCH 00/11] igb: Add experimental VF live migration support Cédric Le Goater
` (2 preceding siblings ...)
2026-07-27 5:39 ` [RFC PATCH 03/11] igb: Add migration BAR with state machine Cédric Le Goater
@ 2026-07-27 5:39 ` Cédric Le Goater
2026-07-27 7:31 ` Akihiko Odaki
2026-07-27 5:39 ` [RFC PATCH 05/11] igb: Add VF post-load fixups " Cédric Le Goater
` (8 subsequent siblings)
12 siblings, 1 reply; 26+ messages in thread
From: Cédric Le Goater @ 2026-07-27 5:39 UTC (permalink / raw)
To: qemu-devel
Cc: Akihiko Odaki, Sriram Yagnaraman, Jason Wang, Alex Williamson,
Michael S . Tsirkin, Peter Xu, Avihai Horon,
Cédric Le Goater
Add igb_pf_get_core() so migration code can reach the PF's IGBCore
from a VF device and implement igb_core_vf_save_state() and
igb_core_vf_load_state() to serialize and restore per-VF device state
through the migration BAR.
The wire format is a versioned blob: header (magic, version, VF
number, register count), offset/value pairs for per-VF registers,
dynamically scanned RA/RA2 entries, and TX context descriptors. PVT
shadow registers (PVTEIMS/PVTEIAC/PVTEIAM) are saved instead of the PF
aggregates which the L1 driver may have transiently cleared.
The load path validates the header, restores registers to mac[], syncs
EITR to eitr_guest_value[], and restores TX context. MSI-X table/PBA
is not saved - L1's VFIO reprograms it after migration.
Assisted-by: Claude
Signed-off-by: Cédric Le Goater <clg@redhat.com>
---
hw/net/igb_core.h | 1 +
hw/net/igb.c | 6 +
hw/net/igb_migration.c | 277 ++++++++++++++++++++++++++++++++++++++++-
3 files changed, 283 insertions(+), 1 deletion(-)
diff --git a/hw/net/igb_core.h b/hw/net/igb_core.h
index d70b54e318f1..58d4f57c99bb 100644
--- a/hw/net/igb_core.h
+++ b/hw/net/igb_core.h
@@ -143,4 +143,5 @@ igb_receive_iov(IGBCore *core, const struct iovec *iov, int iovcnt);
void
igb_start_recv(IGBCore *core);
+IGBCore *igb_pf_get_core(void *pf);
#endif
diff --git a/hw/net/igb.c b/hw/net/igb.c
index b43235996db2..222413dd237a 100644
--- a/hw/net/igb.c
+++ b/hw/net/igb.c
@@ -134,6 +134,12 @@ void igb_vf_reset(void *opaque, uint16_t vfn)
igb_core_vf_reset(&s->core, vfn);
}
+IGBCore *igb_pf_get_core(void *pf)
+{
+ IGBState *s = IGB(pf);
+ return &s->core;
+}
+
static bool
igb_io_get_reg_index(IGBState *s, uint32_t *idx)
{
diff --git a/hw/net/igb_migration.c b/hw/net/igb_migration.c
index a0d044815fd9..61cf155a188d 100644
--- a/hw/net/igb_migration.c
+++ b/hw/net/igb_migration.c
@@ -75,16 +75,226 @@ bool igbvf_add_migration_cap(PCIDevice *dev, Error **errp)
return true;
}
+static IGBCore *igbvf_get_core(IgbVfState *s)
+{
+ return igb_pf_get_core(pcie_sriov_get_pf(PCI_DEVICE(s)));
+}
+
/*
* =====================================================================
* Per-VF state serialization / deserialization
* =====================================================================
+ *
+ * Wire format:
+ * uint32_t magic (IGB_MIG_CAP_MAGIC)
+ * uint32_t version (1)
+ * uint32_t vfn (VF number)
+ * uint32_t num_regs (total register pairs, fixed + RA)
+ * { uint32_t offset; uint32_t value; } regs[num_regs]
+ * uint32_t num_tx_ctx (number of TX queue context blocks)
+ * { raw struct igb_tx data } tx_ctx[num_tx_ctx]
*/
+/* Maximum number of registers in the VF state slice */
+#define IGB_VF_MAX_REGS 128
+
+/* Register offsets that constitute a VF's state slice */
+static void igb_vf_reg_list(uint16_t vfn, uint32_t *offsets, int *count)
+{
+ int n = 0;
+ int q0 = vfn;
+ int q1 = vfn + IGB_NUM_VM_POOLS;
+
+ /* Per-VF control and interrupt registers */
+ offsets[n++] = E1000_PVTCTRL(vfn) >> 2;
+ offsets[n++] = E1000_PVTEICS(vfn) >> 2;
+ offsets[n++] = E1000_PVTEIMS(vfn) >> 2;
+ offsets[n++] = E1000_PVTEIMC(vfn) >> 2;
+ offsets[n++] = E1000_PVTEIAC(vfn) >> 2;
+ offsets[n++] = E1000_PVTEIAM(vfn) >> 2;
+ offsets[n++] = E1000_PVTEICR(vfn) >> 2;
+
+ /* Per-VF statistics */
+ offsets[n++] = E1000_PVFGPRC(vfn) >> 2;
+ offsets[n++] = E1000_PVFGPTC(vfn) >> 2;
+ offsets[n++] = E1000_PVFGORC(vfn) >> 2;
+ offsets[n++] = E1000_PVFGOTC(vfn) >> 2;
+ offsets[n++] = E1000_PVFMPRC(vfn) >> 2;
+ offsets[n++] = E1000_PVFGPRLBC(vfn) >> 2;
+ offsets[n++] = E1000_PVFGPTLBC(vfn) >> 2;
+ offsets[n++] = E1000_PVFGORLBC(vfn) >> 2;
+ offsets[n++] = E1000_PVFGOTLBC(vfn) >> 2;
+
+ /* Mailbox */
+ offsets[n++] = E1000_V2PMAILBOX(vfn) >> 2;
+ offsets[n++] = E1000_P2VMAILBOX(vfn) >> 2;
+
+ /* Per-VF config */
+ offsets[n++] = E1000_VMOLR(vfn) >> 2;
+ offsets[n++] = E1000_VMVIR(vfn) >> 2;
+ offsets[n++] = E1000_PSRTYPE(vfn) >> 2;
+
+ /*
+ * VF receive addresses (RA/RA2) are saved dynamically in
+ * igb_core_vf_save_state by scanning for entries whose pool
+ * bits match this VF - the PF driver chooses the RA slot.
+ */
+
+ /* Interrupt routing */
+ offsets[n++] = (E1000_VTIVAR + vfn * 4) >> 2;
+ offsets[n++] = (E1000_VTIVAR_MISC + vfn * 4) >> 2;
+
+ /*
+ * EITR (Extended Interrupt Throttle Register) - 3 vectors per VF.
+ * Each VF has 3 MSI-X vectors, each with its own EITR controlling
+ * interrupt coalescing. Without saving these, interrupt
+ * throttling resets to zero after migration which can cause
+ * interrupt storms or latency changes. VF N uses PF EITR indices
+ * (22 - N*3) .. (24 - N*3).
+ */
+ {
+ int eitr_base = 22 - vfn * 3;
+ offsets[n++] = E1000_EITR(eitr_base) >> 2;
+ offsets[n++] = E1000_EITR(eitr_base + 1) >> 2;
+ offsets[n++] = E1000_EITR(eitr_base + 2) >> 2;
+ }
+
+ /* RX and TX queue registers for queues q0 and q1 */
+#define ADD_QUEUE_REGS(q) do { \
+ offsets[n++] = E1000_RDBAL(q) >> 2; \
+ offsets[n++] = E1000_RDBAH(q) >> 2; \
+ offsets[n++] = E1000_RDLEN(q) >> 2; \
+ offsets[n++] = E1000_SRRCTL(q) >> 2; \
+ offsets[n++] = E1000_RDH(q) >> 2; \
+ offsets[n++] = E1000_RDT(q) >> 2; \
+ offsets[n++] = E1000_RXDCTL(q) >> 2; \
+ offsets[n++] = E1000_RXCTL(q) >> 2; \
+ offsets[n++] = E1000_RQDPC(q) >> 2; \
+ offsets[n++] = E1000_TDBAL(q) >> 2; \
+ offsets[n++] = E1000_TDBAH(q) >> 2; \
+ offsets[n++] = E1000_TDLEN(q) >> 2; \
+ offsets[n++] = E1000_TDH(q) >> 2; \
+ offsets[n++] = E1000_TDT(q) >> 2; \
+ offsets[n++] = E1000_TXDCTL(q) >> 2; \
+ offsets[n++] = E1000_TXCTL(q) >> 2; \
+ offsets[n++] = E1000_TDWBAL(q) >> 2; \
+ offsets[n++] = E1000_TDWBAH(q) >> 2; \
+} while (0)
+
+ ADD_QUEUE_REGS(q0);
+ ADD_QUEUE_REGS(q1);
+#undef ADD_QUEUE_REGS
+
+ g_assert(n <= IGB_VF_MAX_REGS);
+ *count = n;
+}
+
+/*
+ * Scan RA and RA2 arrays for receive address entries assigned to
+ * this VF. The PF driver picks the RA slot, so we cannot use a
+ * fixed index - instead check each entry's pool bits.
+ */
+static uint32_t *igb_core_vf_save_ra(IGBCore *core, uint16_t vfn,
+ uint32_t *p, int *total_regs)
+{
+ uint32_t vf_pool_bit = E1000_RAH_POOL_1 << vfn;
+ static const struct {
+ uint32_t base;
+ int count;
+ } ra_banks[] = {
+ { RA, 16 },
+ { RA2, 8 },
+ };
+ int i, j;
+
+ for (i = 0; i < ARRAY_SIZE(ra_banks); i++) {
+ for (j = 0; j < ra_banks[i].count; j++) {
+ uint32_t ral_off = ra_banks[i].base + j * 2;
+ uint32_t rah_off = ra_banks[i].base + j * 2 + 1;
+ uint32_t rah_val = core->mac[rah_off];
+
+ if ((rah_val & E1000_RAH_AV) && (rah_val & vf_pool_bit)) {
+ *p++ = cpu_to_le32(ral_off);
+ *p++ = cpu_to_le32(core->mac[ral_off]);
+ *p++ = cpu_to_le32(rah_off);
+ *p++ = cpu_to_le32(rah_val);
+ *total_regs += 2;
+ }
+ }
+ }
+ return p;
+}
+
+static uint32_t *igb_core_vf_save_tx_ctx(IGBCore *core, int queue,
+ uint32_t *p)
+{
+ memcpy(p, &core->tx[queue], sizeof(struct igb_tx));
+ return (uint32_t *)((uint8_t *)p + sizeof(struct igb_tx));
+}
+
+static size_t igb_core_vf_state_max_size(int num_fixed_regs)
+{
+ int max_ra_entries = 16 + 8; /* RA bank (16) + RA2 bank (8) */
+ int max_ra_regs = max_ra_entries * 2; /* RAL + RAH per entry */
+
+ return 4 * sizeof(uint32_t) /* header */
+ + num_fixed_regs * 2 * sizeof(uint32_t) /* fixed reg pairs */
+ + max_ra_regs * 2 * sizeof(uint32_t) /* RA reg pairs */
+ + sizeof(uint32_t) /* num_tx_ctx */
+ + 2 * sizeof(struct igb_tx); /* TX context */
+}
+
static int igb_core_vf_save_state(IgbVfState *s,
void *buf, size_t buf_size)
{
- int size = 0;
+ IGBCore *core = igbvf_get_core(s);
+ uint32_t offsets[IGB_VF_MAX_REGS];
+ int num_regs, total_regs;
+ uint32_t *p = buf;
+ uint32_t *num_regs_p;
+ int i, size;
+ int q0 = s->vfn;
+ int q1 = s->vfn + IGB_NUM_VM_POOLS;
+
+ /*
+ * Save PVT shadow registers (PVTEIMS/PVTEIAC/PVTEIAM) instead of
+ * extracting from PF aggregates - the L1 PF driver may have
+ * transiently cleared EIMS via EIMC. The load path ORs them back.
+ */
+ igb_vf_reg_list(s->vfn, offsets, &num_regs);
+
+ if (!buf) {
+ return igb_core_vf_state_max_size(num_regs);
+ }
+
+ if (igb_core_vf_state_max_size(num_regs) > buf_size) {
+ return -IGB_MIG_ERR_BAD_SIZE;
+ }
+
+ /* Header: magic, version, vfn, num_regs (updated below) */
+ *p++ = cpu_to_le32(IGB_MIG_CAP_MAGIC);
+ *p++ = cpu_to_le32(1); /* version */
+ *p++ = cpu_to_le32(s->vfn);
+ num_regs_p = p;
+ *p++ = cpu_to_le32(num_regs);
+
+ for (i = 0; i < num_regs; i++) {
+ *p++ = cpu_to_le32(offsets[i]);
+ *p++ = cpu_to_le32(core->mac[offsets[i]]);
+ }
+
+ total_regs = num_regs;
+
+ p = igb_core_vf_save_ra(core, s->vfn, p, &total_regs);
+
+ *num_regs_p = cpu_to_le32(total_regs);
+
+ /* TX context descriptors for this VF's two queues */
+ *p++ = cpu_to_le32(2); /* num_tx_ctx */
+ p = igb_core_vf_save_tx_ctx(core, q0, p);
+ p = igb_core_vf_save_tx_ctx(core, q1, p);
+
+ size = (uint8_t *)p - (uint8_t *)buf;
trace_igbvf_mig_save_state(s->vfn, size);
return size;
@@ -98,9 +308,74 @@ static int igb_core_vf_max_data_size(IgbVfState *s)
return size;
}
+static const void *igb_core_vf_load_tx_ctx(IGBCore *core, int queue,
+ const void *data)
+{
+ struct NetTxPkt *saved_pkt = core->tx[queue].tx_pkt;
+
+ memcpy(&core->tx[queue], data, sizeof(struct igb_tx));
+ core->tx[queue].tx_pkt = saved_pkt;
+ return (const uint8_t *)data + sizeof(struct igb_tx);
+}
+
static int igb_core_vf_load_state(IgbVfState *s,
const void *buf, size_t size)
{
+ IGBCore *core = igbvf_get_core(s);
+ const uint32_t *p = buf;
+ uint32_t magic, version, saved_vfn, num_regs, num_tx;
+ int i;
+ int q0 = s->vfn;
+ int q1 = s->vfn + IGB_NUM_VM_POOLS;
+
+ magic = le32_to_cpu(*p++);
+ version = le32_to_cpu(*p++);
+ saved_vfn = le32_to_cpu(*p++);
+ num_regs = le32_to_cpu(*p++);
+
+ if (magic != IGB_MIG_CAP_MAGIC) {
+ return -IGB_MIG_ERR_BAD_MAGIC;
+ }
+ if (version != IGB_MIG_CAP_VERSION) {
+ return -IGB_MIG_ERR_BAD_VERSION;
+ }
+ if (saved_vfn != s->vfn) {
+ return -IGB_MIG_ERR_BAD_VFN;
+ }
+ if (num_regs > IGB_VF_MAX_REGS) {
+ return -IGB_MIG_ERR_BAD_SIZE;
+ }
+
+ for (i = 0; i < num_regs; i++) {
+ uint32_t offset = le32_to_cpu(*p++);
+ uint32_t value = le32_to_cpu(*p++);
+
+ if (offset < E1000E_MAC_SIZE) {
+ core->mac[offset] = value;
+
+ /*
+ * Sync EITR to eitr_guest_value[] shadow array, stripping
+ * E1000_EITR_CNT_IGNR so guest register readback returns
+ * the correct value.
+ */
+ if (offset >= EITR0 && offset < EITR0 + IGB_INTR_NUM) {
+ core->eitr_guest_value[offset - EITR0] =
+ value & ~E1000_EITR_CNT_IGNR;
+ }
+ }
+ }
+
+ num_tx = le32_to_cpu(*p++);
+ if (num_tx == 2) {
+ p = igb_core_vf_load_tx_ctx(core, q0, p);
+ p = igb_core_vf_load_tx_ctx(core, q1, p);
+ }
+
+ /*
+ * MSI-X table/PBA is not saved - L1's VFIO reprograms it with
+ * destination-specific IRTE references after migration.
+ */
+
trace_igbvf_mig_load_state(s->vfn, (uint32_t)size);
return 0;
}
--
2.55.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [RFC PATCH 05/11] igb: Add VF post-load fixups for live migration
2026-07-27 5:39 [RFC PATCH 00/11] igb: Add experimental VF live migration support Cédric Le Goater
` (3 preceding siblings ...)
2026-07-27 5:39 ` [RFC PATCH 04/11] igb: Add VF state serialization for live migration Cédric Le Goater
@ 2026-07-27 5:39 ` Cédric Le Goater
2026-07-27 7:53 ` Akihiko Odaki
2026-07-27 5:39 ` [RFC PATCH 06/11] igb: Add dirty page tracking for IGBVF migration Cédric Le Goater
` (7 subsequent siblings)
12 siblings, 1 reply; 26+ messages in thread
From: Cédric Le Goater @ 2026-07-27 5:39 UTC (permalink / raw)
To: qemu-devel
Cc: Akihiko Odaki, Sriram Yagnaraman, Jason Wang, Alex Williamson,
Michael S . Tsirkin, Peter Xu, Avihai Horon,
Cédric Le Goater
Add post-load fixups in igbvf_mig_load() to propagate VF interrupt
state that the register load path bypasses by writing directly to
mac[] without triggering register handler side effects.
igb_core_vf_propagate_irqs() ORs PVT shadow values back into the PF
aggregates (EIMS/EIAC/EIAM) and clears stale VF bits from EICR.
igb_core_vf_propagate_ivar() re-applies VTIVAR routing to the shared
IVAR0 entries that the L1 PF driver may have overwritten after L0
vmstate restore.
Assisted-by: Claude
Signed-off-by: Cédric Le Goater <clg@redhat.com>
---
hw/net/igb_core.h | 2 ++
hw/net/igb_core.c | 56 ++++++++++++++++++++++++++++++++++++++++++
hw/net/igb_migration.c | 7 ++++++
3 files changed, 65 insertions(+)
diff --git a/hw/net/igb_core.h b/hw/net/igb_core.h
index 58d4f57c99bb..150567eb346d 100644
--- a/hw/net/igb_core.h
+++ b/hw/net/igb_core.h
@@ -144,4 +144,6 @@ void
igb_start_recv(IGBCore *core);
IGBCore *igb_pf_get_core(void *pf);
+void igb_core_vf_propagate_irqs(IGBCore *core, uint16_t vfn);
+void igb_core_vf_propagate_ivar(IGBCore *core, uint16_t vfn);
#endif
diff --git a/hw/net/igb_core.c b/hw/net/igb_core.c
index 45d8fd795b84..2565dd7f96d3 100644
--- a/hw/net/igb_core.c
+++ b/hw/net/igb_core.c
@@ -4550,3 +4550,59 @@ igb_core_post_load(IGBCore *core)
return 0;
}
+
+/*
+ * Propagate VF interrupt state to PF aggregates after loading VF
+ * registers. The load path writes directly to mac[] bypassing the
+ * register handlers that OR VF bits into EIMS/EIAC/EIAM. Also clear
+ * stale VF bits in EICR that may have been set by packets arriving
+ * between PF vmstate restore and VF state load.
+ */
+void igb_core_vf_propagate_irqs(IGBCore *core, uint16_t vfn)
+{
+ uint32_t shift = 22 - vfn * IGBVF_MSIX_VEC_NUM;
+ uint32_t pvt_idx;
+
+ pvt_idx = PVTEIMS0 + vfn * 0x40;
+ core->mac[EIMS] |= (core->mac[pvt_idx] & 0x7) << shift;
+ pvt_idx = PVTEIAC0 + vfn * 0x40;
+ core->mac[EIAC] |= (core->mac[pvt_idx] & 0x7) << shift;
+ pvt_idx = PVTEIAM0 + vfn * 0x40;
+ core->mac[EIAM] |= (core->mac[pvt_idx] & 0x7) << shift;
+
+ core->mac[EICR] &= ~(0x7 << shift);
+}
+
+/*
+ * Re-apply VTIVAR -> IVAR0 interrupt routing. The L1 PF driver
+ * may have overwritten the shared IVAR0 entries with its own
+ * queue routing after L0 vmstate restore.
+ */
+void igb_core_vf_propagate_ivar(IGBCore *core, uint16_t vfn)
+{
+ uint32_t vtivar = core->mac[VTIVAR + vfn];
+ int n;
+ uint8_t ent;
+ uint32_t mask;
+
+ if (vtivar & E1000_IVAR_VALID) {
+ n = igb_ivar_entry_rx(vfn);
+ ent = E1000_IVAR_VALID |
+ (24 - vfn * IGBVF_MSIX_VEC_NUM - (2 - (vtivar & 0x7)));
+ mask = 0xffU << (8 * (n % 4));
+ core->mac[IVAR0 + n / 4] =
+ (core->mac[IVAR0 + n / 4] & ~mask) |
+ ((uint32_t)ent << (8 * (n % 4)));
+ }
+
+ ent = vtivar >> 8;
+ if (ent & E1000_IVAR_VALID) {
+ n = igb_ivar_entry_tx(vfn);
+ ent = E1000_IVAR_VALID |
+ (24 - vfn * IGBVF_MSIX_VEC_NUM - (2 - (ent & 0x7)));
+ mask = 0xffU << (8 * (n % 4));
+ core->mac[IVAR0 + n / 4] =
+ (core->mac[IVAR0 + n / 4] & ~mask) |
+ ((uint32_t)ent << (8 * (n % 4)));
+ }
+}
diff --git a/hw/net/igb_migration.c b/hw/net/igb_migration.c
index 61cf155a188d..4f123df6795d 100644
--- a/hw/net/igb_migration.c
+++ b/hw/net/igb_migration.c
@@ -382,6 +382,7 @@ static int igb_core_vf_load_state(IgbVfState *s,
static int igbvf_mig_load(IgbVfState *s, const void *buf, size_t size)
{
+ IGBCore *core = igbvf_get_core(s);
int ret;
ret = igb_core_vf_load_state(s, buf, size);
@@ -389,6 +390,12 @@ static int igbvf_mig_load(IgbVfState *s, const void *buf, size_t size)
return ret;
}
+ /*
+ * Post-load: sync VF interrupt and routing state to PF aggregates
+ */
+ igb_core_vf_propagate_irqs(core, s->vfn);
+ igb_core_vf_propagate_ivar(core, s->vfn);
+
return 0;
}
--
2.55.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [RFC PATCH 06/11] igb: Add dirty page tracking for IGBVF migration
2026-07-27 5:39 [RFC PATCH 00/11] igb: Add experimental VF live migration support Cédric Le Goater
` (4 preceding siblings ...)
2026-07-27 5:39 ` [RFC PATCH 05/11] igb: Add VF post-load fixups " Cédric Le Goater
@ 2026-07-27 5:39 ` Cédric Le Goater
2026-07-27 8:15 ` Akihiko Odaki
2026-07-27 5:39 ` [RFC PATCH 07/11] igb: Quiesce VFs on STOP and include PF enable state in migration blob Cédric Le Goater
` (6 subsequent siblings)
12 siblings, 1 reply; 26+ messages in thread
From: Cédric Le Goater @ 2026-07-27 5:39 UTC (permalink / raw)
To: qemu-devel
Cc: Akihiko Odaki, Sriram Yagnaraman, Jason Wang, Alex Williamson,
Michael S . Tsirkin, Peter Xu, Avihai Horon,
Cédric Le Goater
Extend the migration BAR with dirty page tracking registers:
0x020 DIRTY_PGSIZE (RW) - dirty tracking page granularity
0x024 DIRTY_CTRL (WO) - 0=DISABLE, 1=ENABLE, 2=QUERY
0x028 DIRTY_RANGE_IOVA_LO (WO) - low 32 bits of tracked range start
0x02C DIRTY_RANGE_IOVA_HI (WO) - high 32 bits of tracked range start
0x030 DIRTY_RANGE_SIZE (WO) - tracked range size in bytes
0x034 DIRTY_BUF_ADDR_LO (WO) - low 32 bits of shared buffer address
0x038 DIRTY_BUF_ADDR_HI (WO) - high 32 bits of shared buffer address
0x03C DIRTY_STATUS (RO) - result of last DIRTY_CTRL
The CAPS register advertises the maximum number of ranges and
supported page sizes. The driver enables tracking on IOVA ranges
via DIRTY_CTRL and queries dirty bitmaps through a shared buffer.
As for state transfers, dirty buffer DMA is performed through the
PF device (pcie_sriov_get_pf).
Each tracked range maintains its own bitmap scoped to its boundaries.
DMA paths (TX/RX data, descriptor writeback) are instrumented to
record the touched pages. This enables the PRE_COPY state where the
driver iterates on dirty pages while the VM continues to run.
Assisted-by: Claude
Signed-off-by: Cédric Le Goater <clg@redhat.com>
---
docs/system/devices/igb-migration.rst | 73 +++++
hw/net/igb_core.h | 2 +
hw/net/igb_migration.h | 82 ++++++
hw/net/igb_core.c | 48 ++--
hw/net/igb_migration.c | 367 +++++++++++++++++++++++++-
hw/net/igbvf.c | 4 +
hw/net/trace-events | 7 +
7 files changed, 562 insertions(+), 21 deletions(-)
diff --git a/docs/system/devices/igb-migration.rst b/docs/system/devices/igb-migration.rst
index 04c44ef072af..4a3a93f01dfc 100644
--- a/docs/system/devices/igb-migration.rst
+++ b/docs/system/devices/igb-migration.rst
@@ -37,6 +37,14 @@ the following register layout::
0x014 DATA_XFER WO Trigger DMA save or DMA load
0x018 DATA_BUF_ADDR_LO WO Low 32 bits of state DMA buffer address
0x01C DATA_BUF_ADDR_HI WO High 32 bits of state DMA buffer address
+ 0x020 DIRTY_PGSIZE RW Dirty tracking page granularity
+ 0x024 DIRTY_CTRL WO 0=DISABLE, 1=ENABLE, 2=QUERY
+ 0x028 DIRTY_RANGE_IOVA_LO WO Low 32 bits of tracked range start
+ 0x02C DIRTY_RANGE_IOVA_HI WO High 32 bits of tracked range start
+ 0x030 DIRTY_RANGE_SIZE WO Tracked range size in bytes
+ 0x034 DIRTY_BUF_ADDR_LO WO Low 32 bits of shared buffer address
+ 0x038 DIRTY_BUF_ADDR_HI WO High 32 bits of shared buffer address
+ 0x03C DIRTY_STATUS RO Result of last DIRTY_CTRL (0=OK, 1-5=error)
State transitions follow the VFIO migration state machine: the driver
writes to ``DEVICE_STATE`` to move between states and reads ``STATUS``
@@ -61,3 +69,68 @@ code identifying the failure::
4 BAD_VFN VF number mismatch (source != destination)
5 DMA_FAILED DMA transfer to/from state buffer failed
6 NO_BUFFER DATA_XFER without buffer address set
+
+Dirty page tracking
+~~~~~~~~~~~~~~~~~~~
+
+The migration interface supports per-VF dirty page tracking, advertised
+by the ``F_DIRTY`` flag in ``CAPS``. This allows the variant driver to
+enter ``PRE_COPY`` state (``DEVICE_STATE`` = 5) while the VM continues
+to run, iterating on dirty pages to reduce the final stop-and-copy
+window.
+
+The device maintains one dirty tracking engine per range, each with its
+own bitmap scoped to the range boundaries. The ``CAPS`` register
+advertises the maximum number of ranges (``max_ranges`` in bits [11:8])
+and supported page sizes (bits [31:12]).
+
+Dirty tracking is controlled through the ``DIRTY_CTRL`` register:
+
+- **ENABLE** (1): the driver programs a tracked range via
+ ``DIRTY_RANGE_IOVA_LO/HI`` + ``DIRTY_RANGE_SIZE`` then writes
+ ``DIRTY_CTRL=ENABLE``. The device allocates a fixed-size bitmap for
+ the range and begins recording pages touched by DMA (TX data, RX
+ data, descriptor writeback). The page granularity is set by
+ ``DIRTY_PGSIZE`` (default 4096). After each ENABLE the driver reads
+ ``DIRTY_STATUS`` to check for errors.
+- **DISABLE** (0): tears down all ranges and stops tracking.
+- **QUERY** (2): the driver writes (IOVA, size, page_size) into a
+ shared buffer registered via ``DIRTY_BUF_ADDR_LO/HI`` (PF DMA
+ address, as for state transfers), then writes
+ ``DIRTY_CTRL=QUERY``. The device finds the matching range, copies
+ the dirty bitmap into the shared buffer, clears the tracked bits,
+ and sets the buffer's completion status.
+
+``DIRTY_STATUS`` values after each ``DIRTY_CTRL`` write::
+
+ 0 OK Success
+ 1 TOO_MANY_RANGES Exceeds max_ranges from CAPS
+ 2 BAD_RANGE Invalid range (zero size)
+ 3 BAD_PGSIZE Invalid or misaligned page size
+ 4 NOT_ENABLED Query without prior enable
+ 5 NO_BUFFER Query without shared buffer
+
+Dirty query shared buffer
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The shared buffer used for dirty queries is registered via
+``DIRTY_BUF_ADDR_LO/HI`` (PF DMA address). It is cache-line aligned
+(64 bytes) to separate driver-written request fields from
+device-written completion fields::
+
+ Offset Field Written by Description
+ 0x00 iova driver Query range start
+ 0x08 size driver Query range size
+ 0x10 page_size driver Page granularity
+ 0x14 flags driver Reserved (must be 0)
+ 0x18 reserved[10] - Pad to 64-byte cache line
+ 0x40 status device 0 = pending, 1 = complete
+ 0x44 bitmap_size device Bytes written to bitmap
+ 0x48 dirty_page_count device Number of set bits in bitmap
+ 0x4C reserved[12] - Pad to 64-byte cache line
+ 0x80 bitmap[] device Dirty page bitmap
+
+The driver fills the request fields, issues ``DIRTY_CTRL=QUERY``, and
+polls ``status`` for completion. The device reads the request, writes
+the dirty bitmap and completion fields via DMA, then sets
+``status = 1``.
diff --git a/hw/net/igb_core.h b/hw/net/igb_core.h
index 150567eb346d..50cee2c7683c 100644
--- a/hw/net/igb_core.h
+++ b/hw/net/igb_core.h
@@ -83,6 +83,8 @@ struct IGBCore {
struct NetTxPkt *tx_pkt;
} tx[IGB_NUM_QUEUES];
+ IGBVfDirtyState vf_dirty[IGB_MAX_VF_FUNCTIONS];
+
struct NetRxPkt *rx_pkt;
bool has_vnet;
diff --git a/hw/net/igb_migration.h b/hw/net/igb_migration.h
index 739a189810b0..a2e7b363cb04 100644
--- a/hw/net/igb_migration.h
+++ b/hw/net/igb_migration.h
@@ -27,6 +27,12 @@
#define IGB_MIG_CAP_F_STATE (1u << 0) /* device state serialization */
#define IGB_MIG_CAP_F_DIRTY (1u << 1) /* dirty page tracking */
+/* MIG_CAPS register layout (read-only, offset 0x008) */
+#define IGB_MIG_CAPS_MAX_RANGES_SHIFT 8
+#define IGB_MIG_CAPS_MAX_RANGES_MASK (0xfu << 8) /* bits [11:8] */
+#define IGB_MIG_CAPS_MAX_RANGES 4
+#define IGB_MIG_CAPS_PGSIZES_MASK 0xfffff000u /* bits [31:12] */
+
#define IGB_MIG_CAP_SIZE 16
#define IGB_MIG_CAP_OFF_MAGIC 4 /* offset within cap for magic field */
#define IGB_MIG_CAP_OFF_BARID 8 /* offset within cap for BAR id */
@@ -49,6 +55,14 @@
#define IGB_MIG_DATA_XFER 0x014
#define IGB_MIG_DATA_BUF_ADDR_LO 0x018
#define IGB_MIG_DATA_BUF_ADDR_HI 0x01C
+#define IGB_MIG_DIRTY_PGSIZE 0x020
+#define IGB_MIG_DIRTY_CTRL 0x024
+#define IGB_MIG_DIRTY_RANGE_IOVA_LO 0x028
+#define IGB_MIG_DIRTY_RANGE_IOVA_HI 0x02C
+#define IGB_MIG_DIRTY_RANGE_SIZE 0x030
+#define IGB_MIG_DIRTY_BUF_ADDR_LO 0x034
+#define IGB_MIG_DIRTY_BUF_ADDR_HI 0x038
+#define IGB_MIG_DIRTY_STATUS 0x03C
/* DEVICE_STATE values - mirrors VFIO migration states */
#define IGB_MIG_STATE_ERROR 0
@@ -75,6 +89,50 @@
#define IGB_MIG_ERR_DMA_FAILED 5
#define IGB_MIG_ERR_NO_BUFFER 6
+/*
+ * DIRTY_CTRL register values. Write one of these to control
+ * the dirty page tracking state machine.
+ */
+#define IGB_MIG_DIRTY_CTRL_DISABLE 0
+#define IGB_MIG_DIRTY_CTRL_ENABLE 1
+#define IGB_MIG_DIRTY_CTRL_QUERY 2 /* query-and-clear */
+
+/* DIRTY_STATUS register values (read-only, cleared on next DIRTY_CTRL write) */
+#define IGB_MIG_DIRTY_STATUS_OK 0
+#define IGB_MIG_DIRTY_STATUS_TOO_MANY_RANGES 1
+#define IGB_MIG_DIRTY_STATUS_BAD_RANGE 2
+#define IGB_MIG_DIRTY_STATUS_BAD_PGSIZE 3
+#define IGB_MIG_DIRTY_STATUS_NOT_ENABLED 4
+#define IGB_MIG_DIRTY_STATUS_NO_BUFFER 5
+
+#define IGB_MIG_DIRTY_DEFAULT_PGSIZE 4096
+
+/*
+ * Dirty query shared buffer layout - DMA between driver and device.
+ * The driver writes request fields, kicks DIRTY_CTRL=QUERY, and the
+ * device reads the request, fills bitmap + completion via DMA. Each
+ * section is cache-line aligned (64 bytes).
+ */
+struct igb_mig_dirty_query {
+ /* Cache line 0: request (written by driver) */
+ uint64_t iova;
+ uint64_t size;
+ uint32_t page_size;
+ uint32_t flags;
+ uint32_t reserved0[10];
+
+ /* Cache line 1: completion (written by device) */
+ uint32_t status;
+ uint32_t bitmap_size;
+ uint32_t dirty_page_count;
+ uint32_t reserved1[12];
+
+ /* Cache line 2+: bitmap (written by device) */
+ uint8_t bitmap[];
+};
+
+#define IGB_MIG_DIRTY_STATUS_COMPLETE 1
+
typedef struct IgbVfMigState {
bool migration_cap;
MemoryRegion mig_bar;
@@ -84,6 +142,12 @@ typedef struct IgbVfMigState {
uint8_t mig_data[IGB_VF_STATE_MAX_SIZE];
uint32_t mig_data_size;
uint64_t mig_data_buf_addr;
+
+ uint32_t mig_dirty_pgsize;
+ uint64_t mig_dirty_range_iova;
+ uint32_t mig_dirty_range_size;
+ uint64_t mig_dirty_buf_addr;
+ uint32_t mig_dirty_status;
} IgbVfMigState;
typedef struct IgbVfState IgbVfState;
@@ -92,4 +156,22 @@ bool igbvf_add_migration_cap(PCIDevice *dev, Error **errp);
void igbvf_mig_bar_init(IgbVfState *s);
void igbvf_mig_state_reset(IgbVfState *s);
+typedef struct IGBVfDirtyRange {
+ uint64_t iova;
+ uint64_t size;
+ uint64_t page_size;
+ unsigned long *bitmap;
+ uint64_t nbits;
+} IGBVfDirtyRange;
+
+typedef struct IGBVfDirtyState {
+ uint32_t num_ranges;
+ IGBVfDirtyRange ranges[IGB_MIG_CAPS_MAX_RANGES];
+} IGBVfDirtyState;
+
+typedef struct IGBCore IGBCore;
+void igb_core_dirty_track_dma(IGBCore *core, int vfn,
+ dma_addr_t addr, dma_addr_t len);
+void igb_core_vf_dirty_disable(IgbVfState *s);
+
#endif
diff --git a/hw/net/igb_core.c b/hw/net/igb_core.c
index 2565dd7f96d3..03c349575c0a 100644
--- a/hw/net/igb_core.c
+++ b/hw/net/igb_core.c
@@ -824,6 +824,14 @@ igb_rx_ring_init(IGBCore *core, E1000E_RxRing *rxr, int idx)
rxr->i = &i[idx];
}
+static inline void
+igb_pci_dma_write(IGBCore *core, PCIDevice *dev, int vfn,
+ dma_addr_t addr, const void *buf, dma_addr_t len)
+{
+ pci_dma_write(dev, addr, buf, len);
+ igb_core_dirty_track_dma(core, vfn, addr, len);
+}
+
static uint32_t
igb_txdesc_writeback(IGBCore *core, dma_addr_t base,
union e1000_adv_tx_desc *tx_desc,
@@ -847,13 +855,15 @@ igb_txdesc_writeback(IGBCore *core, dma_addr_t base,
if (tdwba & 1) {
uint32_t buffer = cpu_to_le32(core->mac[txi->dh]);
- pci_dma_write(d, tdwba & ~3, &buffer, sizeof(buffer));
+ igb_pci_dma_write(core, d, txi->idx % IGB_NUM_VM_POOLS,
+ tdwba & ~3, &buffer, sizeof(buffer));
} else {
uint32_t status = le32_to_cpu(tx_desc->wb.status) | E1000_TXD_STAT_DD;
tx_desc->wb.status = cpu_to_le32(status);
- pci_dma_write(d, base + offsetof(union e1000_adv_tx_desc, wb),
- &tx_desc->wb, sizeof(tx_desc->wb));
+ igb_pci_dma_write(core, d, txi->idx % IGB_NUM_VM_POOLS,
+ base + offsetof(union e1000_adv_tx_desc, wb),
+ &tx_desc->wb, sizeof(tx_desc->wb));
}
return igb_tx_wb_eic(core, txi->idx);
@@ -1264,6 +1274,7 @@ typedef struct IGBPacketRxDMAState {
size_t iov_ofs;
bool do_ps;
bool is_first;
+ int vfn;
IGBBAState bastate;
hwaddr ba[IGB_MAX_PS_BUFFERS];
IGBSplitDescriptorData ps_desc_data;
@@ -1590,7 +1601,8 @@ igb_write_rx_descr(IGBCore *core,
static inline void
igb_pci_dma_write_rx_desc(IGBCore *core, PCIDevice *dev, dma_addr_t addr,
- union e1000_rx_desc_union *desc, dma_addr_t len)
+ union e1000_rx_desc_union *desc, dma_addr_t len,
+ int vfn)
{
if (igb_rx_use_legacy_descriptor(core)) {
struct e1000_rx_desc *d = &desc->legacy;
@@ -1598,11 +1610,12 @@ igb_pci_dma_write_rx_desc(IGBCore *core, PCIDevice *dev, dma_addr_t addr,
uint8_t status = d->status;
d->status &= ~E1000_RXD_STAT_DD;
- pci_dma_write(dev, addr, desc, len);
+ igb_pci_dma_write(core, dev, vfn, addr, desc, len);
if (status & E1000_RXD_STAT_DD) {
d->status = status;
- pci_dma_write(dev, addr + offset, &status, sizeof(status));
+ igb_pci_dma_write(core, dev, vfn,
+ addr + offset, &status, sizeof(status));
}
} else {
union e1000_adv_rx_desc *d = &desc->adv;
@@ -1611,11 +1624,12 @@ igb_pci_dma_write_rx_desc(IGBCore *core, PCIDevice *dev, dma_addr_t addr,
uint32_t status = d->wb.upper.status_error;
d->wb.upper.status_error &= ~E1000_RXD_STAT_DD;
- pci_dma_write(dev, addr, desc, len);
+ igb_pci_dma_write(core, dev, vfn, addr, desc, len);
if (status & E1000_RXD_STAT_DD) {
d->wb.upper.status_error = status;
- pci_dma_write(dev, addr + offset, &status, sizeof(status));
+ igb_pci_dma_write(core, dev, vfn,
+ addr + offset, &status, sizeof(status));
}
}
}
@@ -1737,9 +1751,9 @@ igb_write_hdr_frag_to_rx_buffers(IGBCore *core,
{
assert(data_len <= pdma_st->rx_desc_header_buf_size -
pdma_st->bastate.written[0]);
- pci_dma_write(d,
- pdma_st->ba[0] + pdma_st->bastate.written[0],
- data, data_len);
+ igb_pci_dma_write(core, d, pdma_st->vfn,
+ pdma_st->ba[0] + pdma_st->bastate.written[0],
+ data, data_len);
pdma_st->bastate.written[0] += data_len;
pdma_st->bastate.cur_idx = 1;
}
@@ -1804,10 +1818,10 @@ igb_write_payload_frag_to_rx_buffers(IGBCore *core,
data,
bytes_to_write);
- pci_dma_write(d,
- pdma_st->ba[pdma_st->bastate.cur_idx] +
- pdma_st->bastate.written[pdma_st->bastate.cur_idx],
- data, bytes_to_write);
+ igb_pci_dma_write(core, d, pdma_st->vfn,
+ pdma_st->ba[pdma_st->bastate.cur_idx] +
+ pdma_st->bastate.written[pdma_st->bastate.cur_idx],
+ data, bytes_to_write);
pdma_st->bastate.written[pdma_st->bastate.cur_idx] += bytes_to_write;
data += bytes_to_write;
@@ -1908,6 +1922,7 @@ igb_write_packet_to_guest(IGBCore *core, struct NetRxPkt *pkt,
rxi = rxr->i;
rx_desc_len = core->rx_desc_len;
+ pdma_st.vfn = rxi->idx % IGB_NUM_VM_POOLS;
pdma_st.rx_desc_packet_buf_size = igb_rxbufsize(core, rxi);
pdma_st.rx_desc_header_buf_size = igb_rxhdrbufsize(core, rxi);
pdma_st.iov = net_rx_pkt_get_iovec(pkt);
@@ -1944,7 +1959,8 @@ igb_write_packet_to_guest(IGBCore *core, struct NetRxPkt *pkt,
etqf, ts,
&pdma_st,
rxi);
- igb_pci_dma_write_rx_desc(core, d, base, &desc, rx_desc_len);
+ igb_pci_dma_write_rx_desc(core, d, base, &desc, rx_desc_len,
+ rxi->idx % IGB_NUM_VM_POOLS);
igb_ring_advance(core, rxi, rx_desc_len / E1000_MIN_RX_DESC_LEN);
} while (pdma_st.desc_offset < pdma_st.total_size);
diff --git a/hw/net/igb_migration.c b/hw/net/igb_migration.c
index 4f123df6795d..9a9ce902bb4a 100644
--- a/hw/net/igb_migration.c
+++ b/hw/net/igb_migration.c
@@ -8,6 +8,8 @@
#include "qemu/osdep.h"
#include "qemu/log.h"
+#include "qemu/units.h"
+#include "qemu/bitmap.h"
#include "hw/pci/pci_device.h"
#include "hw/pci/pcie.h"
#include "net/eth.h"
@@ -69,7 +71,7 @@ bool igbvf_add_migration_cap(PCIDevice *dev, Error **errp)
pci_set_long(dev->config + offset + IGB_MIG_CAP_OFF_BARID,
IGB_MIG_BAR_IDX);
pci_set_long(dev->config + offset + IGB_MIG_CAP_OFF_FLAGS,
- IGB_MIG_CAP_F_STATE);
+ IGB_MIG_CAP_F_STATE | IGB_MIG_CAP_F_DIRTY);
trace_igbvf_mig_cap_add(pcie_sriov_vf_number(dev), offset);
return true;
@@ -399,6 +401,161 @@ static int igbvf_mig_load(IgbVfState *s, const void *buf, size_t size)
return 0;
}
+/*
+ * =====================================================================
+ * Per-VF dirty page tracking
+ * =====================================================================
+ *
+ * All VF DMA writes in igb_core.c go through igb_pci_dma_write(),
+ * which calls igb_core_dirty_track_dma() to mark the target page in a
+ * per-range bitmap before performing the actual DMA.
+ *
+ * The IGBCore::vf_dirty[] bitmaps live in IGBCore so they are easily
+ * accessible from the core TX and RX paths without reaching back into
+ * VF state.
+ */
+
+void igb_core_dirty_track_dma(IGBCore *core, int vfn,
+ dma_addr_t addr, dma_addr_t len)
+{
+ IGBVfDirtyState *ds = &core->vf_dirty[vfn];
+ bool matched = false;
+ uint32_t i;
+
+ if (!ds->num_ranges) {
+ return;
+ }
+
+ trace_igb_core_dirty_track_dma(vfn, addr, len);
+
+ for (i = 0; i < ds->num_ranges; i++) {
+ IGBVfDirtyRange *r = &ds->ranges[i];
+ uint64_t r_end = r->iova + r->size;
+ uint64_t dma_end = addr + len;
+ uint64_t start, end, start_page, end_page, page;
+
+ if (addr >= r_end || dma_end <= r->iova) {
+ continue;
+ }
+
+ matched = true;
+ start = MAX(addr, r->iova);
+ end = MIN(dma_end, r_end);
+
+ start_page = (start - r->iova) / r->page_size;
+ end_page = (end - 1 - r->iova) / r->page_size;
+
+ for (page = start_page; page <= end_page; page++) {
+ if (page < r->nbits) {
+ set_bit(page, r->bitmap);
+ }
+ }
+ }
+
+ if (!matched) {
+ trace_igb_core_dirty_track_dma_drop(vfn, addr, len);
+ }
+}
+
+static IGBVfDirtyState *igb_core_vf_dirty_state(IgbVfState *s)
+{
+ IGBCore *core = igbvf_get_core(s);
+ return &core->vf_dirty[s->vfn];
+}
+
+#define IGB_MIG_DIRTY_MAX_PAGES ((256 * GiB) / (4 * KiB))
+
+static uint32_t igb_core_vf_dirty_enable(IgbVfState *s, uint64_t pgsize,
+ uint64_t range_iova, uint64_t range_size)
+{
+ IGBVfDirtyState *ds = igb_core_vf_dirty_state(s);
+ IGBVfDirtyRange *r;
+
+ if (ds->num_ranges >= IGB_MIG_CAPS_MAX_RANGES) {
+ return IGB_MIG_DIRTY_STATUS_TOO_MANY_RANGES;
+ }
+
+ if (!range_size) {
+ return IGB_MIG_DIRTY_STATUS_BAD_RANGE;
+ }
+
+ if (!pgsize || (range_iova % pgsize) || (range_size % pgsize)) {
+ return IGB_MIG_DIRTY_STATUS_BAD_PGSIZE;
+ }
+
+ if (range_size / pgsize > IGB_MIG_DIRTY_MAX_PAGES) {
+ return IGB_MIG_DIRTY_STATUS_BAD_RANGE;
+ }
+
+ r = &ds->ranges[ds->num_ranges];
+ r->iova = range_iova;
+ r->size = range_size;
+ r->page_size = pgsize;
+ r->nbits = range_size / pgsize;
+ r->bitmap = bitmap_new(r->nbits);
+ ds->num_ranges++;
+ return IGB_MIG_DIRTY_STATUS_OK;
+}
+
+void igb_core_vf_dirty_disable(IgbVfState *s)
+{
+ IGBVfDirtyState *ds = igb_core_vf_dirty_state(s);
+ uint32_t i;
+
+ for (i = 0; i < ds->num_ranges; i++) {
+ IGBVfDirtyRange *r = &ds->ranges[i];
+
+ g_free(r->bitmap);
+ r->bitmap = NULL;
+ r->nbits = 0;
+ }
+ ds->num_ranges = 0;
+}
+
+static bool igb_core_vf_dirty_enabled(IgbVfState *s)
+{
+ IGBVfDirtyState *ds = igb_core_vf_dirty_state(s);
+ return !!ds->num_ranges;
+}
+
+static bool igb_core_vf_dirty_query(IgbVfState *s,
+ void *buf, size_t buf_size, size_t *out_size,
+ uint64_t range_iova, uint64_t range_size)
+{
+ IGBVfDirtyState *ds = igb_core_vf_dirty_state(s);
+ uint32_t i;
+
+ for (i = 0; i < ds->num_ranges; i++) {
+ IGBVfDirtyRange *r = &ds->ranges[i];
+ uint64_t start_page, range_pages, count;
+
+ if (range_iova < r->iova ||
+ range_iova + range_size > r->iova + r->size) {
+ continue;
+ }
+
+ start_page = (range_iova - r->iova) / r->page_size;
+ range_pages = (uint64_t)range_size / r->page_size;
+ count = MIN(range_pages, (uint64_t)buf_size * 8);
+
+ memset(buf, 0, buf_size);
+
+ if (start_page < r->nbits) {
+ uint64_t avail = r->nbits - start_page;
+ uint64_t n = MIN(count, avail);
+
+ bitmap_copy_with_src_offset(buf, r->bitmap, start_page, n);
+ bitmap_clear(r->bitmap, start_page, n);
+ }
+
+ *out_size = bitmap_empty(buf, count) ? 0 : DIV_ROUND_UP(count, 8);
+ return true;
+ }
+
+ *out_size = 0;
+ return false;
+}
+
/* ================================================================
* Migration BAR register read/write handlers
* ================================================================ */
@@ -410,25 +567,53 @@ static bool igbvf_mig_set_state(IgbVfState *s, uint32_t new_state)
int ret;
switch (new_state) {
+ case IGB_MIG_STATE_PRE_COPY:
+ if (old != IGB_MIG_STATE_RUNNING) {
+ return false;
+ }
+ /*
+ * Take an initial snapshot so DATA_SIZE reflects the actual
+ * state size and DATA_AVAIL is set for the driver.
+ */
+ ret = igb_core_vf_save_state(s, ms->mig_data, sizeof(ms->mig_data));
+ if (ret < 0) {
+ ms->mig_error = -ret;
+ ms->mig_state = IGB_MIG_STATE_ERROR;
+ return false;
+ }
+ ms->mig_data_size = ret;
+ break;
+
case IGB_MIG_STATE_STOP:
if (old != IGB_MIG_STATE_RUNNING &&
old != IGB_MIG_STATE_STOP_COPY &&
+ old != IGB_MIG_STATE_PRE_COPY &&
old != IGB_MIG_STATE_RESUMING &&
old != IGB_MIG_STATE_ERROR) {
return false;
}
/* Restore DATA_SIZE to max, same as at reset */
ms->mig_data_size = igb_core_vf_max_data_size(s);
+ if (old == IGB_MIG_STATE_PRE_COPY ||
+ old == IGB_MIG_STATE_STOP_COPY ||
+ old == IGB_MIG_STATE_ERROR) {
+ igb_core_vf_dirty_disable(s);
+ }
break;
case IGB_MIG_STATE_RUNNING:
- if (old != IGB_MIG_STATE_STOP) {
+ if (old != IGB_MIG_STATE_STOP &&
+ old != IGB_MIG_STATE_PRE_COPY) {
return false;
}
+ if (old == IGB_MIG_STATE_PRE_COPY) {
+ igb_core_vf_dirty_disable(s);
+ }
break;
case IGB_MIG_STATE_STOP_COPY:
- if (old != IGB_MIG_STATE_STOP) {
+ if (old != IGB_MIG_STATE_STOP &&
+ old != IGB_MIG_STATE_PRE_COPY) {
return false;
}
ret = igb_core_vf_save_state(s, ms->mig_data, sizeof(ms->mig_data));
@@ -466,7 +651,8 @@ static uint32_t igbvf_mig_get_status(IgbVfState *s)
if (ms->mig_state == IGB_MIG_STATE_ERROR) {
status |= IGB_MIG_STATUS_ERR(ms->mig_error);
}
- if (ms->mig_state == IGB_MIG_STATE_STOP_COPY && ms->mig_data_size > 0) {
+ if ((ms->mig_state == IGB_MIG_STATE_STOP_COPY ||
+ ms->mig_state == IGB_MIG_STATE_PRE_COPY) && ms->mig_data_size > 0) {
status |= IGB_MIG_STATUS_DATA_AVAIL;
}
@@ -486,6 +672,30 @@ static void igbvf_mig_data_xfer(IgbVfState *s, uint32_t val)
}
switch (ms->mig_state) {
+ case IGB_MIG_STATE_PRE_COPY:
+ /*
+ * Re-snapshot device state so the driver can read a fresh
+ * copy on each pre-copy iteration.
+ */
+ ret = igb_core_vf_save_state(s, ms->mig_data, sizeof(ms->mig_data));
+ if (ret < 0) {
+ ms->mig_error = -ret;
+ ms->mig_state = IGB_MIG_STATE_ERROR;
+ return;
+ }
+ ms->mig_data_size = ret;
+ r = pci_dma_write(pcie_sriov_get_pf(PCI_DEVICE(s)),
+ ms->mig_data_buf_addr,
+ ms->mig_data, ms->mig_data_size);
+ if (r != MEMTX_OK) {
+ qemu_log_mask(LOG_GUEST_ERROR,
+ "igbvf: VF%u state write failed at 0x%" PRIx64 "\n",
+ s->vfn, ms->mig_data_buf_addr);
+ ms->mig_error = IGB_MIG_ERR_DMA_FAILED;
+ ms->mig_state = IGB_MIG_STATE_ERROR;
+ }
+ break;
+
case IGB_MIG_STATE_STOP_COPY:
/* Save: DMA-write serialized state to driver buffer */
r = pci_dma_write(pcie_sriov_get_pf(PCI_DEVICE(s)),
@@ -547,7 +757,9 @@ static uint64_t igbvf_mig_read(void *opaque, hwaddr addr, unsigned size)
val = igbvf_mig_get_status(s);
break;
case IGB_MIG_CAPS:
- val = IGB_MIG_CAP_F_STATE;
+ val = IGB_MIG_CAP_F_STATE | IGB_MIG_CAP_F_DIRTY |
+ (IGB_MIG_CAPS_MAX_RANGES << IGB_MIG_CAPS_MAX_RANGES_SHIFT) |
+ (1u << 12); /* 4K page size supported */
break;
case IGB_MIG_VERSION:
val = IGB_MIG_CAP_VERSION;
@@ -555,6 +767,19 @@ static uint64_t igbvf_mig_read(void *opaque, hwaddr addr, unsigned size)
case IGB_MIG_DATA_SIZE:
val = ms->mig_data_size;
break;
+ case IGB_MIG_DIRTY_PGSIZE:
+ val = ms->mig_dirty_pgsize ? ms->mig_dirty_pgsize
+ : IGB_MIG_DIRTY_DEFAULT_PGSIZE;
+ break;
+ case IGB_MIG_DIRTY_BUF_ADDR_LO:
+ val = (uint32_t)ms->mig_dirty_buf_addr;
+ break;
+ case IGB_MIG_DIRTY_BUF_ADDR_HI:
+ val = (uint32_t)(ms->mig_dirty_buf_addr >> 32);
+ break;
+ case IGB_MIG_DIRTY_STATUS:
+ val = ms->mig_dirty_status;
+ break;
default:
qemu_log_mask(LOG_GUEST_ERROR,
"igbvf: VF%u bad migration BAR read at 0x%"
@@ -567,6 +792,106 @@ static uint64_t igbvf_mig_read(void *opaque, hwaddr addr, unsigned size)
return val;
}
+static uint32_t igbvf_mig_dirty_count(const void *bitmap, size_t size)
+{
+ const unsigned long *p = bitmap;
+ size_t n = size / sizeof(unsigned long);
+ uint32_t count = 0;
+
+ for (size_t i = 0; i < n; i++) {
+ count += ctpopl(p[i]);
+ }
+ return count;
+}
+
+static void igbvf_mig_dirty_query(IgbVfState *s, uint64_t pgsize)
+{
+ IgbVfMigState *ms = &s->mig;
+ PCIDevice *dev = pcie_sriov_get_pf(PCI_DEVICE(s));
+ uint64_t buf_addr = ms->mig_dirty_buf_addr;
+ uint64_t range_iova = 0, range_size = 0;
+ uint32_t bmp_bytes;
+ size_t out_size;
+ g_autofree void *bitmap = NULL;
+ bool valid;
+
+ ldq_le_pci_dma(dev,
+ buf_addr + offsetof(struct igb_mig_dirty_query, iova),
+ &range_iova, MEMTXATTRS_UNSPECIFIED);
+ ldq_le_pci_dma(dev,
+ buf_addr + offsetof(struct igb_mig_dirty_query, size),
+ &range_size, MEMTXATTRS_UNSPECIFIED);
+
+ bmp_bytes = DIV_ROUND_UP(range_size / pgsize, 8);
+ bitmap = g_malloc0(bmp_bytes);
+
+ valid = igb_core_vf_dirty_query(s, bitmap, bmp_bytes, &out_size,
+ range_iova, range_size);
+
+ if (valid && out_size) {
+ if (pci_dma_write(dev,
+ buf_addr + offsetof(struct igb_mig_dirty_query, bitmap),
+ bitmap, out_size)) {
+ qemu_log_mask(LOG_GUEST_ERROR,
+ "igbvf: VF%u dirty bitmap write failed at 0x%" PRIx64 "\n",
+ s->vfn, buf_addr);
+ valid = false;
+ }
+ }
+
+ uint32_t dirty_pages = valid ? igbvf_mig_dirty_count(bitmap, out_size) : 0;
+
+ stl_le_pci_dma(dev,
+ buf_addr + offsetof(struct igb_mig_dirty_query, bitmap_size),
+ valid ? out_size : 0, MEMTXATTRS_UNSPECIFIED);
+ stl_le_pci_dma(dev,
+ buf_addr + offsetof(struct igb_mig_dirty_query, dirty_page_count),
+ dirty_pages, MEMTXATTRS_UNSPECIFIED);
+ stl_le_pci_dma(dev,
+ buf_addr + offsetof(struct igb_mig_dirty_query, status),
+ valid ? IGB_MIG_DIRTY_STATUS_COMPLETE : 0,
+ MEMTXATTRS_UNSPECIFIED);
+
+ trace_igbvf_mig_dirty_query(s->vfn, (uint64_t)out_size, dirty_pages);
+}
+
+static void igbvf_mig_dirty_ctrl(IgbVfState *s, uint32_t val)
+{
+ IgbVfMigState *ms = &s->mig;
+ uint64_t pgsize = ms->mig_dirty_pgsize
+ ? ms->mig_dirty_pgsize : IGB_MIG_DIRTY_DEFAULT_PGSIZE;
+
+ switch (val) {
+ case IGB_MIG_DIRTY_CTRL_ENABLE:
+ ms->mig_dirty_status = igb_core_vf_dirty_enable(s, pgsize,
+ ms->mig_dirty_range_iova,
+ ms->mig_dirty_range_size);
+ if (ms->mig_dirty_status) {
+ break;
+ }
+ trace_igbvf_mig_dirty_enable(s->vfn, pgsize,
+ ms->mig_dirty_range_size / pgsize);
+ break;
+ case IGB_MIG_DIRTY_CTRL_DISABLE:
+ igb_core_vf_dirty_disable(s);
+ ms->mig_dirty_status = IGB_MIG_DIRTY_STATUS_OK;
+ trace_igbvf_mig_dirty_disable(s->vfn);
+ break;
+ case IGB_MIG_DIRTY_CTRL_QUERY:
+ if (!igb_core_vf_dirty_enabled(s)) {
+ ms->mig_dirty_status = IGB_MIG_DIRTY_STATUS_NOT_ENABLED;
+ break;
+ }
+ if (!ms->mig_dirty_buf_addr) {
+ ms->mig_dirty_status = IGB_MIG_DIRTY_STATUS_NO_BUFFER;
+ break;
+ }
+ igbvf_mig_dirty_query(s, pgsize);
+ ms->mig_dirty_status = IGB_MIG_DIRTY_STATUS_OK;
+ break;
+ }
+}
+
static void igbvf_mig_write(void *opaque, hwaddr addr, uint64_t val,
unsigned size)
{
@@ -599,6 +924,31 @@ static void igbvf_mig_write(void *opaque, hwaddr addr, uint64_t val,
ms->mig_data_buf_addr =
deposit64(ms->mig_data_buf_addr, 32, 32, val);
break;
+ case IGB_MIG_DIRTY_PGSIZE:
+ ms->mig_dirty_pgsize = (uint32_t)val;
+ break;
+ case IGB_MIG_DIRTY_CTRL:
+ igbvf_mig_dirty_ctrl(s, (uint32_t)val);
+ break;
+ case IGB_MIG_DIRTY_RANGE_IOVA_LO:
+ ms->mig_dirty_range_iova =
+ deposit64(ms->mig_dirty_range_iova, 0, 32, val);
+ break;
+ case IGB_MIG_DIRTY_RANGE_IOVA_HI:
+ ms->mig_dirty_range_iova =
+ deposit64(ms->mig_dirty_range_iova, 32, 32, val);
+ break;
+ case IGB_MIG_DIRTY_RANGE_SIZE:
+ ms->mig_dirty_range_size = (uint32_t)val;
+ break;
+ case IGB_MIG_DIRTY_BUF_ADDR_LO:
+ ms->mig_dirty_buf_addr =
+ deposit64(ms->mig_dirty_buf_addr, 0, 32, val);
+ break;
+ case IGB_MIG_DIRTY_BUF_ADDR_HI:
+ ms->mig_dirty_buf_addr =
+ deposit64(ms->mig_dirty_buf_addr, 32, 32, val);
+ break;
default:
qemu_log_mask(LOG_GUEST_ERROR,
"igbvf: VF%u bad migration BAR write at 0x%"
@@ -643,5 +993,12 @@ void igbvf_mig_state_reset(IgbVfState *s)
ms->mig_data_size = igb_core_vf_max_data_size(s);
ms->mig_data_buf_addr = 0;
memset(ms->mig_data, 0, sizeof(ms->mig_data));
+
+ igb_core_vf_dirty_disable(s);
+ ms->mig_dirty_pgsize = 0;
+ ms->mig_dirty_range_iova = 0;
+ ms->mig_dirty_range_size = 0;
+ ms->mig_dirty_buf_addr = 0;
+ ms->mig_dirty_status = IGB_MIG_DIRTY_STATUS_OK;
trace_igbvf_mig_reset(s->vfn);
}
diff --git a/hw/net/igbvf.c b/hw/net/igbvf.c
index e9f9fc3369d8..398306d3bfc6 100644
--- a/hw/net/igbvf.c
+++ b/hw/net/igbvf.c
@@ -306,6 +306,10 @@ static void igbvf_pci_uninit(PCIDevice *dev)
{
IgbVfState *s = IGBVF(dev);
+ if (s->mig.migration_cap) {
+ igb_core_vf_dirty_disable(s);
+ }
+
pcie_aer_exit(dev);
pcie_cap_exit(dev);
msix_unuse_all_vectors(dev);
diff --git a/hw/net/trace-events b/hw/net/trace-events
index 0b13a99b3f32..95fdea89d49c 100644
--- a/hw/net/trace-events
+++ b/hw/net/trace-events
@@ -303,6 +303,13 @@ igbvf_mig_set_state_err(uint16_t vfn, uint32_t old_state, uint32_t new_state) "V
igbvf_mig_save_state(uint16_t vfn, uint32_t size) "VF%u: saved %u bytes of device state"
igbvf_mig_load_state(uint16_t vfn, uint32_t size) "VF%u: loaded %u bytes of device state"
igbvf_mig_reset(uint16_t vfn) "VF%u: migration state reset"
+igbvf_mig_dirty_enable(uint16_t vfn, uint64_t pgsize, uint64_t nbits) "VF%u: dirty tracking enabled pgsize=%"PRIu64" nbits=%"PRIu64
+igbvf_mig_dirty_disable(uint16_t vfn) "VF%u: dirty tracking disabled"
+igbvf_mig_dirty_query(uint16_t vfn, uint64_t size, uint32_t dirty_pages) "VF%u: dirty query returned %"PRIu64" bytes, %u dirty pages"
+
+# igb_core.c - VF migration diagnostics
+igb_core_dirty_track_dma(int vfn, uint64_t addr, uint64_t len) "VF%d: dirty DMA addr=0x%"PRIx64" len=%"PRIu64
+igb_core_dirty_track_dma_drop(int vfn, uint64_t addr, uint64_t len) "VF%d: dirty DMA dropped addr=0x%"PRIx64" len=%"PRIu64" no matching range"
# spapr_llan.c
spapr_vlan_get_rx_bd_from_pool_found(int pool, int32_t count, uint32_t rx_bufs) "pool=%d count=%"PRId32" rxbufs=%"PRIu32
--
2.55.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [RFC PATCH 07/11] igb: Quiesce VFs on STOP and include PF enable state in migration blob
2026-07-27 5:39 [RFC PATCH 00/11] igb: Add experimental VF live migration support Cédric Le Goater
` (5 preceding siblings ...)
2026-07-27 5:39 ` [RFC PATCH 06/11] igb: Add dirty page tracking for IGBVF migration Cédric Le Goater
@ 2026-07-27 5:39 ` Cédric Le Goater
2026-07-27 8:19 ` Akihiko Odaki
2026-07-27 5:39 ` [RFC PATCH 08/11] igb: Fix post-migration RX ring deadlock Cédric Le Goater
` (5 subsequent siblings)
12 siblings, 1 reply; 26+ messages in thread
From: Cédric Le Goater @ 2026-07-27 5:39 UTC (permalink / raw)
To: qemu-devel
Cc: Akihiko Odaki, Sriram Yagnaraman, Jason Wang, Alex Williamson,
Michael S . Tsirkin, Peter Xu, Avihai Horon,
Cédric Le Goater
Quiesce VF queues when entering the STOP state by disabling RX and TX
at the PF level (VFRE/VFTE). This ensures no DMA activity occurs while
the device state is being serialized. The unquiesce path re-enables
queues when returning to RUNNING.
The VFRE and VFTE registers are shared PF-level registers with one bit
per VF controlling RX and TX enable respectively. Writing the full
32-bit register during restore would clobber other VFs' bits, so these
bits cannot be included in the regular per-VF register list. Instead,
save the per-VF VFRE/VFTE bit state at quiesce time - before the bits
are cleared - and append it to the migration blob as two bytes. On the
destination, extract these bits and restore them during unquiesce.
For QEMU's emulated igb, the device state machine guarantees no DMA in
these states. This register establishes the protocol for real hardware
implementations where DMA drain has latency.
Assisted-by: Claude
Signed-off-by: Cédric Le Goater <clg@redhat.com>
---
hw/net/igb_migration.h | 4 ++
hw/net/igb_migration.c | 110 +++++++++++++++++++++++++++++++++++++++--
hw/net/trace-events | 6 ++-
3 files changed, 115 insertions(+), 5 deletions(-)
diff --git a/hw/net/igb_migration.h b/hw/net/igb_migration.h
index a2e7b363cb04..414412f6392e 100644
--- a/hw/net/igb_migration.h
+++ b/hw/net/igb_migration.h
@@ -75,6 +75,7 @@
/* MIG_STATUS bits */
#define IGB_MIG_STATUS_DATA_AVAIL (1u << 0)
#define IGB_MIG_STATUS_ERROR (1u << 1)
+#define IGB_MIG_STATUS_QUIESCED (1u << 2)
/* MIG_STATUS error codes in bits [15:8], valid when ERROR bit is set */
#define IGB_MIG_STATUS_ERR_SHIFT 8
@@ -143,6 +144,9 @@ typedef struct IgbVfMigState {
uint32_t mig_data_size;
uint64_t mig_data_buf_addr;
+ bool mig_saved_vfre;
+ bool mig_saved_vfte;
+
uint32_t mig_dirty_pgsize;
uint64_t mig_dirty_range_iova;
uint32_t mig_dirty_range_size;
diff --git a/hw/net/igb_migration.c b/hw/net/igb_migration.c
index 9a9ce902bb4a..187077c14c48 100644
--- a/hw/net/igb_migration.c
+++ b/hw/net/igb_migration.c
@@ -95,6 +95,7 @@ static IGBCore *igbvf_get_core(IgbVfState *s)
* { uint32_t offset; uint32_t value; } regs[num_regs]
* uint32_t num_tx_ctx (number of TX queue context blocks)
* { raw struct igb_tx data } tx_ctx[num_tx_ctx]
+ * uint32_t vfre_vfte (VFRE in [7:0], VFTE in [15:8])
*/
/* Maximum number of registers in the VF state slice */
@@ -227,6 +228,13 @@ static uint32_t *igb_core_vf_save_ra(IGBCore *core, uint16_t vfn,
return p;
}
+static uint32_t *igb_core_vf_save_vfre_vfte(uint32_t *p,
+ bool vfre, bool vfte)
+{
+ *p++ = cpu_to_le32((uint32_t)vfre | ((uint32_t)vfte << 8));
+ return p;
+}
+
static uint32_t *igb_core_vf_save_tx_ctx(IGBCore *core, int queue,
uint32_t *p)
{
@@ -243,12 +251,14 @@ static size_t igb_core_vf_state_max_size(int num_fixed_regs)
+ num_fixed_regs * 2 * sizeof(uint32_t) /* fixed reg pairs */
+ max_ra_regs * 2 * sizeof(uint32_t) /* RA reg pairs */
+ sizeof(uint32_t) /* num_tx_ctx */
- + 2 * sizeof(struct igb_tx); /* TX context */
+ + 2 * sizeof(struct igb_tx) /* TX context */
+ + sizeof(uint32_t); /* VFRE + VFTE */
}
static int igb_core_vf_save_state(IgbVfState *s,
void *buf, size_t buf_size)
{
+ IgbVfMigState *ms = &s->mig;
IGBCore *core = igbvf_get_core(s);
uint32_t offsets[IGB_VF_MAX_REGS];
int num_regs, total_regs;
@@ -296,9 +306,15 @@ static int igb_core_vf_save_state(IgbVfState *s,
p = igb_core_vf_save_tx_ctx(core, q0, p);
p = igb_core_vf_save_tx_ctx(core, q1, p);
+ /* Per-VF VFRE/VFTE enable bits */
+ p = igb_core_vf_save_vfre_vfte(p, ms->mig_saved_vfre,
+ ms->mig_saved_vfte);
+
size = (uint8_t *)p - (uint8_t *)buf;
- trace_igbvf_mig_save_state(s->vfn, size);
+ trace_igbvf_mig_save_state(s->vfn, size, ms->mig_saved_vfre,
+ ms->mig_saved_vfte,
+ core->mac[VFRE]);
return size;
}
@@ -310,6 +326,16 @@ static int igb_core_vf_max_data_size(IgbVfState *s)
return size;
}
+static const void *igb_core_vf_load_vfre_vfte(const void *p,
+ bool *out_vfre, bool *out_vfte)
+{
+ uint32_t val = le32_to_cpu(*(const uint32_t *)p);
+
+ *out_vfre = !!(val & 0xff);
+ *out_vfte = !!((val >> 8) & 0xff);
+ return (const uint8_t *)p + sizeof(uint32_t);
+}
+
static const void *igb_core_vf_load_tx_ctx(IGBCore *core, int queue,
const void *data)
{
@@ -323,6 +349,7 @@ static const void *igb_core_vf_load_tx_ctx(IGBCore *core, int queue,
static int igb_core_vf_load_state(IgbVfState *s,
const void *buf, size_t size)
{
+ IgbVfMigState *ms = &s->mig;
IGBCore *core = igbvf_get_core(s);
const uint32_t *p = buf;
uint32_t magic, version, saved_vfn, num_regs, num_tx;
@@ -378,7 +405,13 @@ static int igb_core_vf_load_state(IgbVfState *s,
* destination-specific IRTE references after migration.
*/
- trace_igbvf_mig_load_state(s->vfn, (uint32_t)size);
+ /* Per-VF VFRE/VFTE enable bits */
+ p = igb_core_vf_load_vfre_vfte(p, &ms->mig_saved_vfre,
+ &ms->mig_saved_vfte);
+
+ trace_igbvf_mig_load_state(s->vfn, (uint32_t)size,
+ ms->mig_saved_vfre,
+ ms->mig_saved_vfte);
return 0;
}
@@ -387,6 +420,12 @@ static int igbvf_mig_load(IgbVfState *s, const void *buf, size_t size)
IGBCore *core = igbvf_get_core(s);
int ret;
+ /*
+ * Pre-load: Clear the VFLRE bit before restoring state so the PF
+ * watchdog does not overwrite what we are about to load.
+ */
+ core->mac[VFLRE] &= ~BIT(s->vfn);
+
ret = igb_core_vf_load_state(s, buf, size);
if (ret < 0) {
return ret;
@@ -556,6 +595,45 @@ static bool igb_core_vf_dirty_query(IgbVfState *s,
return false;
}
+/* Quiesce a VF by disabling its RX and TX at the PF level. */
+static void igb_core_vf_quiesce(IgbVfState *s)
+{
+ IgbVfMigState *ms = &s->mig;
+ IGBCore *core = igbvf_get_core(s);
+
+ ms->mig_saved_vfre = !!(core->mac[VFRE] & BIT(s->vfn));
+ ms->mig_saved_vfte = !!(core->mac[VFTE] & BIT(s->vfn));
+
+ core->mac[VFRE] &= ~BIT(s->vfn);
+ core->mac[VFTE] &= ~BIT(s->vfn);
+ trace_igbvf_mig_quiesce(s->vfn, core->mac[VFRE], core->mac[VFTE]);
+}
+
+static void igb_core_vf_unquiesce(IgbVfState *s)
+{
+ IgbVfMigState *ms = &s->mig;
+ IGBCore *core = igbvf_get_core(s);
+ bool re = ms->mig_saved_vfre;
+ bool te = ms->mig_saved_vfte;
+
+ if (re) {
+ core->mac[VFRE] |= BIT(s->vfn);
+ } else {
+ core->mac[VFRE] &= ~BIT(s->vfn);
+ }
+ if (te) {
+ core->mac[VFTE] |= BIT(s->vfn);
+ } else {
+ core->mac[VFTE] &= ~BIT(s->vfn);
+ }
+
+ trace_igbvf_mig_unquiesce(s->vfn, core->mac[VFRE], core->mac[VFTE]);
+
+ if (re) {
+ igb_start_recv(core);
+ }
+}
+
/* ================================================================
* Migration BAR register read/write handlers
* ================================================================ */
@@ -599,6 +677,10 @@ static bool igbvf_mig_set_state(IgbVfState *s, uint32_t new_state)
old == IGB_MIG_STATE_ERROR) {
igb_core_vf_dirty_disable(s);
}
+ if (old == IGB_MIG_STATE_RUNNING ||
+ old == IGB_MIG_STATE_PRE_COPY) {
+ igb_core_vf_quiesce(s);
+ }
break;
case IGB_MIG_STATE_RUNNING:
@@ -609,6 +691,9 @@ static bool igbvf_mig_set_state(IgbVfState *s, uint32_t new_state)
if (old == IGB_MIG_STATE_PRE_COPY) {
igb_core_vf_dirty_disable(s);
}
+ if (old == IGB_MIG_STATE_STOP) {
+ igb_core_vf_unquiesce(s);
+ }
break;
case IGB_MIG_STATE_STOP_COPY:
@@ -616,6 +701,9 @@ static bool igbvf_mig_set_state(IgbVfState *s, uint32_t new_state)
old != IGB_MIG_STATE_PRE_COPY) {
return false;
}
+ if (old == IGB_MIG_STATE_PRE_COPY) {
+ igb_core_vf_quiesce(s);
+ }
ret = igb_core_vf_save_state(s, ms->mig_data, sizeof(ms->mig_data));
if (ret < 0) {
ms->mig_error = -ret;
@@ -656,6 +744,20 @@ static uint32_t igbvf_mig_get_status(IgbVfState *s)
status |= IGB_MIG_STATUS_DATA_AVAIL;
}
+ /*
+ * QUIESCED tells the driver it is safe to read device state.
+ * STOP and STOP_COPY must guarantee that no VF DMA is in flight;
+ * igb_core_vf_quiesce() enforces this by disabling RX/TX for the
+ * VF.
+ *
+ * Under QEMU, DMA completes synchronously within the MMIO handler
+ * so there is nothing to drain, but real hardware would need this.
+ */
+ if (ms->mig_state == IGB_MIG_STATE_STOP ||
+ ms->mig_state == IGB_MIG_STATE_STOP_COPY) {
+ status |= IGB_MIG_STATUS_QUIESCED;
+ }
+
return status;
}
@@ -1000,5 +1102,7 @@ void igbvf_mig_state_reset(IgbVfState *s)
ms->mig_dirty_range_size = 0;
ms->mig_dirty_buf_addr = 0;
ms->mig_dirty_status = IGB_MIG_DIRTY_STATUS_OK;
+ ms->mig_saved_vfre = true;
+ ms->mig_saved_vfte = true;
trace_igbvf_mig_reset(s->vfn);
}
diff --git a/hw/net/trace-events b/hw/net/trace-events
index 95fdea89d49c..f4940e3e176d 100644
--- a/hw/net/trace-events
+++ b/hw/net/trace-events
@@ -300,8 +300,8 @@ igbvf_mig_bar_read(uint16_t vfn, uint64_t addr, uint64_t val) "VF%u: BAR read ad
igbvf_mig_bar_write(uint16_t vfn, uint64_t addr, uint64_t val) "VF%u: BAR write addr=0x%"PRIx64" val=0x%"PRIx64
igbvf_mig_set_state(uint16_t vfn, uint32_t old_state, uint32_t new_state) "VF%u: state %u -> %u"
igbvf_mig_set_state_err(uint16_t vfn, uint32_t old_state, uint32_t new_state) "VF%u: invalid transition %u -> %u"
-igbvf_mig_save_state(uint16_t vfn, uint32_t size) "VF%u: saved %u bytes of device state"
-igbvf_mig_load_state(uint16_t vfn, uint32_t size) "VF%u: loaded %u bytes of device state"
+igbvf_mig_save_state(uint16_t vfn, int size, bool vfre, bool vfte, uint32_t reg_vfre) "VF%u: saved %d bytes vfre=%d vfte=%d VFRE=0x%x"
+igbvf_mig_load_state(uint16_t vfn, uint32_t size, bool vfre, bool vfte) "VF%u: loaded %u bytes vfre=%d vfte=%d"
igbvf_mig_reset(uint16_t vfn) "VF%u: migration state reset"
igbvf_mig_dirty_enable(uint16_t vfn, uint64_t pgsize, uint64_t nbits) "VF%u: dirty tracking enabled pgsize=%"PRIu64" nbits=%"PRIu64
igbvf_mig_dirty_disable(uint16_t vfn) "VF%u: dirty tracking disabled"
@@ -310,6 +310,8 @@ igbvf_mig_dirty_query(uint16_t vfn, uint64_t size, uint32_t dirty_pages) "VF%u:
# igb_core.c - VF migration diagnostics
igb_core_dirty_track_dma(int vfn, uint64_t addr, uint64_t len) "VF%d: dirty DMA addr=0x%"PRIx64" len=%"PRIu64
igb_core_dirty_track_dma_drop(int vfn, uint64_t addr, uint64_t len) "VF%d: dirty DMA dropped addr=0x%"PRIx64" len=%"PRIu64" no matching range"
+igbvf_mig_quiesce(uint16_t vfn, uint32_t vfre, uint32_t vfte) "VF%u: quiesce VFRE=0x%x VFTE=0x%x"
+igbvf_mig_unquiesce(uint16_t vfn, uint32_t vfre, uint32_t vfte) "VF%u: unquiesce VFRE=0x%x VFTE=0x%x"
# spapr_llan.c
spapr_vlan_get_rx_bd_from_pool_found(int pool, int32_t count, uint32_t rx_bufs) "pool=%d count=%"PRId32" rxbufs=%"PRIu32
--
2.55.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [RFC PATCH 08/11] igb: Fix post-migration RX ring deadlock
2026-07-27 5:39 [RFC PATCH 00/11] igb: Add experimental VF live migration support Cédric Le Goater
` (6 preceding siblings ...)
2026-07-27 5:39 ` [RFC PATCH 07/11] igb: Quiesce VFs on STOP and include PF enable state in migration blob Cédric Le Goater
@ 2026-07-27 5:39 ` Cédric Le Goater
2026-07-27 9:36 ` Akihiko Odaki
2026-07-27 5:39 ` [RFC PATCH 09/11] igb: Send RARP after VF migration to update bridge FDB Cédric Le Goater
` (4 subsequent siblings)
12 siblings, 1 reply; 26+ messages in thread
From: Cédric Le Goater @ 2026-07-27 5:39 UTC (permalink / raw)
To: qemu-devel
Cc: Akihiko Odaki, Sriram Yagnaraman, Jason Wang, Alex Williamson,
Michael S . Tsirkin, Peter Xu, Avihai Horon,
Cédric Le Goater
After migration, the VF unquiesce path calls igb_start_recv() to kick
RX processing. This does not trigger an interrupt, so NAPI stays idle
and the RX ring is never replenished.
Replace igb_start_recv() with igb_core_vf_rearm_irqs(): re-apply the
VF's PVT shadow register into the PF aggregates, then raise interrupt
causes via igb_set_eics(). This follows the normal EICR -> EIMS ->
MSI-X routing. NAPI wakes, replenishes the ring via
igbvf_alloc_rx_buffers(), and writes RDT - breaking the deadlock.
The PVT re-application is needed because the L1 PF driver's normal
interrupt handling may have cleared EIMS between state load and
unquiesce.
Assisted-by: Claude
Signed-off-by: Cédric Le Goater <clg@redhat.com>
---
hw/net/igb_core.h | 1 +
hw/net/igb_core.c | 12 ++++++++++++
hw/net/igb_migration.c | 2 +-
3 files changed, 14 insertions(+), 1 deletion(-)
diff --git a/hw/net/igb_core.h b/hw/net/igb_core.h
index 50cee2c7683c..3db520048732 100644
--- a/hw/net/igb_core.h
+++ b/hw/net/igb_core.h
@@ -147,5 +147,6 @@ igb_start_recv(IGBCore *core);
IGBCore *igb_pf_get_core(void *pf);
void igb_core_vf_propagate_irqs(IGBCore *core, uint16_t vfn);
+void igb_core_vf_rearm_irqs(IGBCore *core, uint16_t vfn);
void igb_core_vf_propagate_ivar(IGBCore *core, uint16_t vfn);
#endif
diff --git a/hw/net/igb_core.c b/hw/net/igb_core.c
index 03c349575c0a..1621ee6602d5 100644
--- a/hw/net/igb_core.c
+++ b/hw/net/igb_core.c
@@ -4589,6 +4589,18 @@ void igb_core_vf_propagate_irqs(IGBCore *core, uint16_t vfn)
core->mac[EICR] &= ~(0x7 << shift);
}
+/*
+ * Re-apply VF interrupt enables to PF aggregates and raise interrupt
+ * causes to wake NAPI so it replenishes the ring after migration.
+ */
+void igb_core_vf_rearm_irqs(IGBCore *core, uint16_t vfn)
+{
+ uint32_t shift = 22 - vfn * IGBVF_MSIX_VEC_NUM;
+
+ igb_core_vf_propagate_irqs(core, vfn);
+ igb_set_eics(core, EICS, 0x7 << shift);
+}
+
/*
* Re-apply VTIVAR -> IVAR0 interrupt routing. The L1 PF driver
* may have overwritten the shared IVAR0 entries with its own
diff --git a/hw/net/igb_migration.c b/hw/net/igb_migration.c
index 187077c14c48..a0b4a52c865f 100644
--- a/hw/net/igb_migration.c
+++ b/hw/net/igb_migration.c
@@ -630,7 +630,7 @@ static void igb_core_vf_unquiesce(IgbVfState *s)
trace_igbvf_mig_unquiesce(s->vfn, core->mac[VFRE], core->mac[VFTE]);
if (re) {
- igb_start_recv(core);
+ igb_core_vf_rearm_irqs(core, s->vfn);
}
}
--
2.55.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [RFC PATCH 09/11] igb: Send RARP after VF migration to update bridge FDB
2026-07-27 5:39 [RFC PATCH 00/11] igb: Add experimental VF live migration support Cédric Le Goater
` (7 preceding siblings ...)
2026-07-27 5:39 ` [RFC PATCH 08/11] igb: Fix post-migration RX ring deadlock Cédric Le Goater
@ 2026-07-27 5:39 ` Cédric Le Goater
2026-07-27 10:33 ` Akihiko Odaki
2026-07-27 5:39 ` [RFC PATCH 10/11] docs: Add igb VF migration testing setup guide Cédric Le Goater
` (3 subsequent siblings)
12 siblings, 1 reply; 26+ messages in thread
From: Cédric Le Goater @ 2026-07-27 5:39 UTC (permalink / raw)
To: qemu-devel
Cc: Akihiko Odaki, Sriram Yagnaraman, Jason Wang, Alex Williamson,
Michael S . Tsirkin, Peter Xu, Avihai Horon,
Cédric Le Goater
After VF migration the L0 bridge FDB (Forwarding Database) still maps
the VF's MAC to the old tap port, causing a ~30 s connectivity stall
while peer ARP caches expire.
Add igb_core_vf_get_mac() to look up a VF's MAC address from the PF's
Receive Address registers, and use it to send a RARP broadcast from
the PF's network backend during VF unquiesce so the bridge relearns
the correct port immediately.
Assisted-by: Claude
Signed-off-by: Cédric Le Goater <clg@redhat.com>
---
hw/net/igb_core.h | 1 +
hw/net/igb_core.c | 28 ++++++++++++++++++++++++++++
hw/net/igb_migration.c | 33 +++++++++++++++++++++++++++++++++
hw/net/trace-events | 2 ++
4 files changed, 64 insertions(+)
diff --git a/hw/net/igb_core.h b/hw/net/igb_core.h
index 3db520048732..07b4270708c4 100644
--- a/hw/net/igb_core.h
+++ b/hw/net/igb_core.h
@@ -149,4 +149,5 @@ IGBCore *igb_pf_get_core(void *pf);
void igb_core_vf_propagate_irqs(IGBCore *core, uint16_t vfn);
void igb_core_vf_rearm_irqs(IGBCore *core, uint16_t vfn);
void igb_core_vf_propagate_ivar(IGBCore *core, uint16_t vfn);
+bool igb_core_vf_get_mac(IGBCore *core, uint16_t vfn, uint8_t *mac);
#endif
diff --git a/hw/net/igb_core.c b/hw/net/igb_core.c
index 1621ee6602d5..281f3a474392 100644
--- a/hw/net/igb_core.c
+++ b/hw/net/igb_core.c
@@ -4634,3 +4634,31 @@ void igb_core_vf_propagate_ivar(IGBCore *core, uint16_t vfn)
((uint32_t)ent << (8 * (n % 4)));
}
}
+
+bool igb_core_vf_get_mac(IGBCore *core, uint16_t vfn, uint8_t *mac)
+{
+ uint32_t vf_pool_bit = E1000_RAH_POOL_1 << vfn;
+ static const struct {
+ uint32_t base;
+ int count;
+ } ra_banks[] = {
+ { RA, 16 },
+ { RA2, 8 },
+ };
+ int i, j;
+
+ for (i = 0; i < ARRAY_SIZE(ra_banks); i++) {
+ for (j = 0; j < ra_banks[i].count; j++) {
+ uint32_t ral_off = ra_banks[i].base + j * 2;
+ uint32_t rah_off = ra_banks[i].base + j * 2 + 1;
+ uint32_t rah_val = core->mac[rah_off];
+
+ if ((rah_val & E1000_RAH_AV) && (rah_val & vf_pool_bit)) {
+ stl_le_p(mac, core->mac[ral_off]);
+ stw_le_p(mac + 4, rah_val & 0xffff);
+ return true;
+ }
+ }
+ }
+ return false;
+}
diff --git a/hw/net/igb_migration.c b/hw/net/igb_migration.c
index a0b4a52c865f..fce661f68e3d 100644
--- a/hw/net/igb_migration.c
+++ b/hw/net/igb_migration.c
@@ -609,6 +609,36 @@ static void igb_core_vf_quiesce(IgbVfState *s)
trace_igbvf_mig_quiesce(s->vfn, core->mac[VFRE], core->mac[VFTE]);
}
+/*
+ * Send a RARP broadcast so the network bridge relearns which port
+ * carries this VF's MAC after migration.
+ */
+static void igb_core_vf_send_rarp(IGBCore *core, uint16_t vfn)
+{
+ uint8_t mac[ETH_ALEN];
+ uint8_t buf[60];
+
+ if (!igb_core_vf_get_mac(core, vfn, mac)) {
+ trace_igbvf_mig_no_mac(vfn);
+ return;
+ }
+
+ trace_igbvf_mig_send_rarp(vfn, mac[0], mac[1], mac[2],
+ mac[3], mac[4], mac[5]);
+
+ memset(buf, 0xff, ETH_ALEN);
+ memcpy(buf + 6, mac, ETH_ALEN);
+ stw_be_p(buf + 12, 0x8035); /* ETH_P_RARP */
+ stw_be_p(buf + 14, 1); /* hw addr space: ethernet */
+ stw_be_p(buf + 16, ETH_P_IP); /* protocol addr space */
+ buf[18] = 6; buf[19] = 4; /* hw/proto addr lengths */
+ stw_be_p(buf + 20, 3); /* opcode: RARP request */
+ memcpy(buf + 22, mac, ETH_ALEN);
+ memset(buf + 28, 0, 32);
+
+ qemu_send_packet_raw(qemu_get_queue(core->owner_nic), buf, sizeof(buf));
+}
+
static void igb_core_vf_unquiesce(IgbVfState *s)
{
IgbVfMigState *ms = &s->mig;
@@ -632,6 +662,9 @@ static void igb_core_vf_unquiesce(IgbVfState *s)
if (re) {
igb_core_vf_rearm_irqs(core, s->vfn);
}
+
+ /* TODO : RARP should be sent only if resumed */
+ igb_core_vf_send_rarp(core, s->vfn);
}
/* ================================================================
diff --git a/hw/net/trace-events b/hw/net/trace-events
index f4940e3e176d..1e38d9ab3697 100644
--- a/hw/net/trace-events
+++ b/hw/net/trace-events
@@ -312,6 +312,8 @@ igb_core_dirty_track_dma(int vfn, uint64_t addr, uint64_t len) "VF%d: dirty DMA
igb_core_dirty_track_dma_drop(int vfn, uint64_t addr, uint64_t len) "VF%d: dirty DMA dropped addr=0x%"PRIx64" len=%"PRIu64" no matching range"
igbvf_mig_quiesce(uint16_t vfn, uint32_t vfre, uint32_t vfte) "VF%u: quiesce VFRE=0x%x VFTE=0x%x"
igbvf_mig_unquiesce(uint16_t vfn, uint32_t vfre, uint32_t vfte) "VF%u: unquiesce VFRE=0x%x VFTE=0x%x"
+igbvf_mig_no_mac(uint16_t vfn) "VF%u: no MAC address found, skipping RARP"
+igbvf_mig_send_rarp(uint16_t vfn, uint8_t b0, uint8_t b1, uint8_t b2, uint8_t b3, uint8_t b4, uint8_t b5) "VF%u: RARP %02x:%02x:%02x:%02x:%02x:%02x"
# spapr_llan.c
spapr_vlan_get_rx_bd_from_pool_found(int pool, int32_t count, uint32_t rx_bufs) "pool=%d count=%"PRId32" rxbufs=%"PRIu32
--
2.55.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [RFC PATCH 10/11] docs: Add igb VF migration testing setup guide
2026-07-27 5:39 [RFC PATCH 00/11] igb: Add experimental VF live migration support Cédric Le Goater
` (8 preceding siblings ...)
2026-07-27 5:39 ` [RFC PATCH 09/11] igb: Send RARP after VF migration to update bridge FDB Cédric Le Goater
@ 2026-07-27 5:39 ` Cédric Le Goater
2026-07-27 10:54 ` Akihiko Odaki
2026-07-27 5:39 ` [RFC PATCH 11/11] igb: Add migration statistics registers to VF migration BAR Cédric Le Goater
` (2 subsequent siblings)
12 siblings, 1 reply; 26+ messages in thread
From: Cédric Le Goater @ 2026-07-27 5:39 UTC (permalink / raw)
To: qemu-devel
Cc: Akihiko Odaki, Sriram Yagnaraman, Jason Wang, Alex Williamson,
Michael S . Tsirkin, Peter Xu, Avihai Horon,
Cédric Le Goater
Document the NetworkManager configuration and network topology needed
to test igb VF live migration in a nested virtualization environment.
Signed-off-by: Cédric Le Goater <clg@redhat.com>
---
docs/system/devices/igb-migration.rst | 155 ++++++++++++++++++++++++++
1 file changed, 155 insertions(+)
diff --git a/docs/system/devices/igb-migration.rst b/docs/system/devices/igb-migration.rst
index 4a3a93f01dfc..0e34e4b9fc68 100644
--- a/docs/system/devices/igb-migration.rst
+++ b/docs/system/devices/igb-migration.rst
@@ -134,3 +134,158 @@ The driver fills the request fields, issues ``DIRTY_CTRL=QUERY``, and
polls ``status`` for completion. The device reads the request, writes
the dirty bitmap and completion fields via DMA, then sets
``status = 1``.
+
+Testing setup
+~~~~~~~~~~~~~
+
+The target scenario is nested virtualization::
+
+ L0 QEMU
+ igb PF with x-vf-migration=on
+ └── VFs with migration BAR + vendor cap
+
+ L1 kernel
+ igb-vfio-pci variant driver
+ translates VFIO migration v2 ioctls → BAR2 MMIO
+
+ L1 QEMU (stock, unmodified)
+ vfio-pci device model, standard migration fd
+
+ L2 guest
+ standard igbvf driver, unaware of migration
+
+NetworkManager configuration
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In a nested setup, the L1 VMs (source and destination) have emulated
+igb PFs connected to the L0 bridge. By default, NetworkManager
+acquires DHCP leases on those PF interfaces and on any igb VFs created
+later. This causes the VF MAC address to be learned on the L0 bridge,
+which can misdirect iperf3 traffic after migration.
+
+To prevent this, configure NetworkManager on **both L1 VMs** and on the
+**L2 guest disk image**.
+
+L1 VMs (source and destination)
+...............................
+
+1. Prevent NetworkManager from managing igbvf interfaces:
+
+.. code-block:: bash
+
+ cat > /etc/NetworkManager/conf.d/99-no-igbvf.conf <<EOF
+ [keyfile]
+ unmanaged-devices=driver:igbvf
+ EOF
+
+2. Disable IP on the igb PF connections (keep the interfaces UP for
+ bridging, but with no DHCP lease):
+
+.. code-block:: bash
+
+ # Identify the NM connections for the igb PFs (NOT the virtio management NIC)
+ nmcli -t -f NAME,DEVICE connection show
+
+ # For each igb PF connection:
+ nmcli connection modify "<igb-pf-connection>" ipv4.method disabled ipv6.method disabled
+
+3. Reload NetworkManager:
+
+.. code-block:: bash
+
+ nmcli general reload
+
+L2 guest disk image
+...................
+
+Use ``virt-customize`` to add the igbvf unmanaged config to the guest
+image (offline, before any test run):
+
+.. code-block:: bash
+
+ virt-customize -a /srv/migration/rhel10.qcow2 \
+ --write /etc/NetworkManager/conf.d/99-no-igbvf.conf:'[keyfile]
+ unmanaged-devices=driver:igbvf'
+
+Network diagram
+^^^^^^^^^^^^^^^
+
+The diagram below shows the nested setup where the source and
+destination hosts are themselves VMs (L1) running on a physical
+host (L0) that emulates the igb NIC::
+
+ ┌────────────────────────────────────────────────────────────────────────────┐
+ │ L0: physical host │
+ │ │
+ │ virbr0 192.168.199.1/24 │
+ │ ├── NFS server: /srv/migration │
+ │ └── iperf3 client: iperf3 -c 192.168.199.200 -t 60 -i 1 │
+ │ │ │
+ │ │ L0 virbr0 bridge (192.168.199.0/24) │
+ │ ────┼──────────┬──────────────────┬────────────────────────────── │
+ │ │ │ │ │
+ │ │ ┌────┴────┐ ┌────┴────┐ │
+ │ │ │ virtio │ │ emulated│ │
+ │ │ │c0:ff:ee:│ │ igb PF │ L0 QEMU (vm6) │
+ │ │ │ :00:06 │ │ + igbvf │ tracks DMA dirty pages │
+ │ │ └────┬────┘ └────┬────┘ │
+ │ │ │ │ │
+ │ ┌───┼──────────┼──────────────────┼──────────────────────────────────┐ │
+ │ │ │ L1: vm6 (source) │ │ │
+ │ │ │ enp1s0: 192.168.199.6 │ │ │
+ │ │ │ (management) │ │ │
+ │ │ │ enp8s0 (igb PF, no IP) │ │
+ │ │ │ │ │ │
+ │ │ │ igb VF0 ──► igb-vfio-pci (VFIO) │ │
+ │ │ │ │ dirty_sync → L0 igbvf │ │
+ │ │ │ │ │ │
+ │ │ │ virbr0 │ VFIO passthrough │ │
+ │ │ │ 192.168.200.1/24 │ │ │
+ │ │ │ │ │ │ │
+ │ │ │ ┌────┼─────────────────────┼───────────────────────────┐ │ │
+ │ │ │ │ │ L2: rhel10 guest │ │ │ │
+ │ │ │ │ │ │ │ │ │
+ │ │ │ │ virtio NIC igb VF (enp7s0) │ │ │
+ │ │ │ │ 192.168.200.130/24 192.168.199.200/24 │ │ │
+ │ │ │ │ (SSH login) (iperf3 data path) │ │ │
+ │ │ │ │ │ │ │ │
+ │ │ │ │ iperf3 -s -D │ (listens on 0.0.0.0) │ │ │
+ │ │ │ └──────────────────────────┼───────────────────────────┘ │ │
+ │ │ │ │ │ │
+ │ │ │ virsh migrate --live ──────┼──────────────────► vm7 │ │
+ │ │ │ │ │ │
+ │ └───┼─────────────────────────────┼──────────────────────────────────┘ │
+ │ │ │ │
+ │ │ iperf3 traffic │ │
+ │ └─────────────────────────────┘ │
+ │ │
+ │ ──────────────────────────────────────────────────────────────── │
+ │ │ │ │
+ │ │ ┌────────┐ │ ┌─────────┐ │
+ │ │ │ virtio │ │ │emulated │ L0 QEMU (vm7) │
+ │ │ │c0:ff:ee│ │ │ igb PF │ │
+ │ │ │ :00:07 │ │ │ + igbvf │ │
+ │ │ └────┬───┘ │ └────┬────┘ │
+ │ ┌──────────────┼───────┼────────┼────────────────────────────────────┐ │
+ │ │ L1: vm7 (destination) │ │ │
+ │ │ enp1s0: 192.168.199.7 │ │ │
+ │ │ (management) enp8s0 (igb PF, no IP) │ │
+ │ │ │ │ │
+ │ │ igb VF0 ──► igb-vfio-pci (VFIO) │ │
+ │ │ │ │ │
+ │ │ virbr0 │ VFIO passthrough │ │
+ │ │ 192.168.200.1/24 │ │ │
+ │ │ │ │ │ │
+ │ │ ┌────┼───────────────────────┼────────────────────────────┐ │ │
+ │ │ │ │ L2: rhel10 (after migration) │ │ │
+ │ │ │ │ │ │ │ │
+ │ │ │ virtio NIC igb VF (enp7s0) │ │ │
+ │ │ │ 192.168.200.130/24 192.168.199.200/24 │ │ │
+ │ │ │ │ │ │ │
+ │ │ │ iperf3 -s -D │ (connection survives) │ │ │
+ │ │ └────────────────────────────┼────────────────────────────┘ │ │
+ │ └───────────────────────────────┼────────────────────────────────────┘ │
+ │ │ │
+ │ iperf3 traffic resumes ─────┘ │
+ │ (same IP, same MAC, same L2 segment → transparent to client) │
+ └────────────────────────────────────────────────────────────────────────────┘
--
2.55.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [RFC PATCH 11/11] igb: Add migration statistics registers to VF migration BAR
2026-07-27 5:39 [RFC PATCH 00/11] igb: Add experimental VF live migration support Cédric Le Goater
` (9 preceding siblings ...)
2026-07-27 5:39 ` [RFC PATCH 10/11] docs: Add igb VF migration testing setup guide Cédric Le Goater
@ 2026-07-27 5:39 ` Cédric Le Goater
2026-07-27 11:05 ` Akihiko Odaki
2026-07-27 17:35 ` [RFC PATCH 00/11] igb: Add experimental VF live migration support Cédric Le Goater
2026-07-27 20:36 ` Alex Williamson
12 siblings, 1 reply; 26+ messages in thread
From: Cédric Le Goater @ 2026-07-27 5:39 UTC (permalink / raw)
To: qemu-devel
Cc: Akihiko Odaki, Sriram Yagnaraman, Jason Wang, Alex Williamson,
Michael S . Tsirkin, Peter Xu, Avihai Horon,
Cédric Le Goater
Add read-only statistics registers to the migration BAR at 0x100-0x11C
for monitoring dirty page tracking and DMA activity per VF.
The dma_writes counter is also reported in the dirty query buffer so
the driver gets it alongside the bitmap without an extra MMIO read.
Counters are reset on first DIRTY_CTRL=ENABLE or device reset, so the
driver can read final values after DIRTY_CTRL=DISABLE.
Signed-off-by: Cédric Le Goater <clg@redhat.com>
---
docs/system/devices/igb-migration.rst | 24 ++++++++++-
hw/net/igb_core.h | 1 +
hw/net/igb_migration.h | 22 +++++++++-
hw/net/igb_migration.c | 59 +++++++++++++++++++++++++--
hw/net/trace-events | 2 +-
5 files changed, 102 insertions(+), 6 deletions(-)
diff --git a/docs/system/devices/igb-migration.rst b/docs/system/devices/igb-migration.rst
index 0e34e4b9fc68..3f861eed59cd 100644
--- a/docs/system/devices/igb-migration.rst
+++ b/docs/system/devices/igb-migration.rst
@@ -127,7 +127,9 @@ device-written completion fields::
0x40 status device 0 = pending, 1 = complete
0x44 bitmap_size device Bytes written to bitmap
0x48 dirty_page_count device Number of set bits in bitmap
- 0x4C reserved[12] - Pad to 64-byte cache line
+ 0x4C reserved - Padding
+ 0x50 dma_writes device Total DMA writes since enable
+ 0x58 reserved[10] - Pad to 64-byte cache line
0x80 bitmap[] device Dirty page bitmap
The driver fills the request fields, issues ``DIRTY_CTRL=QUERY``, and
@@ -135,6 +137,26 @@ polls ``status`` for completion. The device reads the request, writes
the dirty bitmap and completion fields via DMA, then sets
``status = 1``.
+Migration statistics
+~~~~~~~~~~~~~~~~~~~~
+
+The migration BAR includes read-only statistics counters in a dedicated
+aperture at offset ``0x100``. These counters provide a live
+per-migration-cycle view of activity without requiring a
+``DIRTY_CTRL=QUERY``::
+
+ Offset Name Description
+ 0x100 MIG_STAT_DMA_WRITES Total DMA write operations tracked
+ 0x104 MIG_STAT_DMA_BYTES_LO Total DMA bytes written (low 32 bits)
+ 0x108 MIG_STAT_DMA_BYTES_HI Total DMA bytes written (high 32 bits)
+ 0x10C MIG_STAT_DIRTY_PAGES_SET Dirty pages marked since enable
+ 0x110 MIG_STAT_DIRTY_PAGES_CLR Dirty pages cleared by queries
+ 0x114 MIG_STAT_DIRTY_PAGE_COUNT Current dirty pages (set minus cleared)
+ 0x118 MIG_STAT_DIRTY_QUERY_CNT Number of QUERY operations
+
+All counters are reset on first ``DIRTY_CTRL=ENABLE`` or device reset,
+so the driver can read final values after ``DIRTY_CTRL=DISABLE``.
+
Testing setup
~~~~~~~~~~~~~
diff --git a/hw/net/igb_core.h b/hw/net/igb_core.h
index 07b4270708c4..91796aec8d7a 100644
--- a/hw/net/igb_core.h
+++ b/hw/net/igb_core.h
@@ -84,6 +84,7 @@ struct IGBCore {
} tx[IGB_NUM_QUEUES];
IGBVfDirtyState vf_dirty[IGB_MAX_VF_FUNCTIONS];
+ IgbVfMigStats vf_mig_stats[IGB_MAX_VF_FUNCTIONS];
struct NetRxPkt *rx_pkt;
diff --git a/hw/net/igb_migration.h b/hw/net/igb_migration.h
index 414412f6392e..6d2e3442781f 100644
--- a/hw/net/igb_migration.h
+++ b/hw/net/igb_migration.h
@@ -64,6 +64,15 @@
#define IGB_MIG_DIRTY_BUF_ADDR_HI 0x038
#define IGB_MIG_DIRTY_STATUS 0x03C
+/* Migration statistics registers (read-only) */
+#define IGB_MIG_STAT_DMA_WRITES 0x100
+#define IGB_MIG_STAT_DMA_BYTES_LO 0x104
+#define IGB_MIG_STAT_DMA_BYTES_HI 0x108
+#define IGB_MIG_STAT_DIRTY_PAGES_SET 0x10C
+#define IGB_MIG_STAT_DIRTY_PAGES_CLR 0x110
+#define IGB_MIG_STAT_DIRTY_PAGE_COUNT 0x114
+#define IGB_MIG_STAT_DIRTY_QUERY_CNT 0x118
+
/* DEVICE_STATE values - mirrors VFIO migration states */
#define IGB_MIG_STATE_ERROR 0
#define IGB_MIG_STATE_STOP 1
@@ -126,7 +135,9 @@ struct igb_mig_dirty_query {
uint32_t status;
uint32_t bitmap_size;
uint32_t dirty_page_count;
- uint32_t reserved1[12];
+ uint32_t reserved1;
+ uint64_t dma_writes;
+ uint32_t reserved2[10];
/* Cache line 2+: bitmap (written by device) */
uint8_t bitmap[];
@@ -154,6 +165,15 @@ typedef struct IgbVfMigState {
uint32_t mig_dirty_status;
} IgbVfMigState;
+typedef struct IgbVfMigStats {
+ uint32_t dma_writes;
+ uint64_t dma_bytes;
+ uint32_t dirty_pages_set;
+ uint32_t dirty_pages_cleared;
+ uint32_t dirty_page_count;
+ uint32_t dirty_query_count;
+} IgbVfMigStats;
+
typedef struct IgbVfState IgbVfState;
void igb_pf_init_migration_bar(PCIDevice *dev);
bool igbvf_add_migration_cap(PCIDevice *dev, Error **errp);
diff --git a/hw/net/igb_migration.c b/hw/net/igb_migration.c
index fce661f68e3d..7fce8d535909 100644
--- a/hw/net/igb_migration.c
+++ b/hw/net/igb_migration.c
@@ -458,6 +458,7 @@ void igb_core_dirty_track_dma(IGBCore *core, int vfn,
dma_addr_t addr, dma_addr_t len)
{
IGBVfDirtyState *ds = &core->vf_dirty[vfn];
+ IgbVfMigStats *stats = &core->vf_mig_stats[vfn];
bool matched = false;
uint32_t i;
@@ -465,6 +466,8 @@ void igb_core_dirty_track_dma(IGBCore *core, int vfn,
return;
}
+ stats->dma_writes++;
+ stats->dma_bytes += len;
trace_igb_core_dirty_track_dma(vfn, addr, len);
for (i = 0; i < ds->num_ranges; i++) {
@@ -486,7 +489,10 @@ void igb_core_dirty_track_dma(IGBCore *core, int vfn,
for (page = start_page; page <= end_page; page++) {
if (page < r->nbits) {
- set_bit(page, r->bitmap);
+ if (!test_and_set_bit(page, r->bitmap)) {
+ stats->dirty_pages_set++;
+ stats->dirty_page_count++;
+ }
}
}
}
@@ -510,6 +516,15 @@ static uint32_t igb_core_vf_dirty_enable(IgbVfState *s, uint64_t pgsize,
IGBVfDirtyState *ds = igb_core_vf_dirty_state(s);
IGBVfDirtyRange *r;
+ /*
+ * Reset stats on first enable so the driver can read them after
+ * disable
+ */
+ if (ds->num_ranges == 0) {
+ memset(&igbvf_get_core(s)->vf_mig_stats[s->vfn], 0,
+ sizeof(IgbVfMigStats));
+ }
+
if (ds->num_ranges >= IGB_MIG_CAPS_MAX_RANGES) {
return IGB_MIG_DIRTY_STATUS_TOO_MANY_RANGES;
}
@@ -561,7 +576,9 @@ static bool igb_core_vf_dirty_query(IgbVfState *s,
void *buf, size_t buf_size, size_t *out_size,
uint64_t range_iova, uint64_t range_size)
{
- IGBVfDirtyState *ds = igb_core_vf_dirty_state(s);
+ IGBCore *core = igbvf_get_core(s);
+ IGBVfDirtyState *ds = &core->vf_dirty[s->vfn];
+ IgbVfMigStats *stats = &core->vf_mig_stats[s->vfn];
uint32_t i;
for (i = 0; i < ds->num_ranges; i++) {
@@ -587,6 +604,13 @@ static bool igb_core_vf_dirty_query(IgbVfState *s,
bitmap_clear(r->bitmap, start_page, n);
}
+ {
+ uint32_t cleared = bitmap_count_one(buf, count);
+ stats->dirty_pages_cleared += cleared;
+ stats->dirty_page_count -= MIN(stats->dirty_page_count, cleared);
+ stats->dirty_query_count++;
+ }
+
*out_size = bitmap_empty(buf, count) ? 0 : DIV_ROUND_UP(count, 8);
return true;
}
@@ -881,6 +905,7 @@ static void igbvf_mig_data_xfer(IgbVfState *s, uint32_t val)
static uint64_t igbvf_mig_read(void *opaque, hwaddr addr, unsigned size)
{
IgbVfState *s = opaque;
+ IgbVfMigStats *stats = &igbvf_get_core(s)->vf_mig_stats[s->vfn];
IgbVfMigState *ms = &s->mig;
uint64_t val = 0;
@@ -915,6 +940,27 @@ static uint64_t igbvf_mig_read(void *opaque, hwaddr addr, unsigned size)
case IGB_MIG_DIRTY_STATUS:
val = ms->mig_dirty_status;
break;
+ case IGB_MIG_STAT_DMA_WRITES:
+ val = stats->dma_writes;
+ break;
+ case IGB_MIG_STAT_DMA_BYTES_LO:
+ val = (uint32_t)stats->dma_bytes;
+ break;
+ case IGB_MIG_STAT_DMA_BYTES_HI:
+ val = (uint32_t)(stats->dma_bytes >> 32);
+ break;
+ case IGB_MIG_STAT_DIRTY_PAGES_SET:
+ val = stats->dirty_pages_set;
+ break;
+ case IGB_MIG_STAT_DIRTY_PAGES_CLR:
+ val = stats->dirty_pages_cleared;
+ break;
+ case IGB_MIG_STAT_DIRTY_PAGE_COUNT:
+ val = stats->dirty_page_count;
+ break;
+ case IGB_MIG_STAT_DIRTY_QUERY_CNT:
+ val = stats->dirty_query_count;
+ break;
default:
qemu_log_mask(LOG_GUEST_ERROR,
"igbvf: VF%u bad migration BAR read at 0x%"
@@ -942,6 +988,7 @@ static uint32_t igbvf_mig_dirty_count(const void *bitmap, size_t size)
static void igbvf_mig_dirty_query(IgbVfState *s, uint64_t pgsize)
{
IgbVfMigState *ms = &s->mig;
+ IgbVfMigStats *stats = &igbvf_get_core(s)->vf_mig_stats[s->vfn];
PCIDevice *dev = pcie_sriov_get_pf(PCI_DEVICE(s));
uint64_t buf_addr = ms->mig_dirty_buf_addr;
uint64_t range_iova = 0, range_size = 0;
@@ -982,12 +1029,17 @@ static void igbvf_mig_dirty_query(IgbVfState *s, uint64_t pgsize)
stl_le_pci_dma(dev,
buf_addr + offsetof(struct igb_mig_dirty_query, dirty_page_count),
dirty_pages, MEMTXATTRS_UNSPECIFIED);
+ stq_le_pci_dma(dev,
+ buf_addr + offsetof(struct igb_mig_dirty_query, dma_writes),
+ stats->dma_writes,
+ MEMTXATTRS_UNSPECIFIED);
stl_le_pci_dma(dev,
buf_addr + offsetof(struct igb_mig_dirty_query, status),
valid ? IGB_MIG_DIRTY_STATUS_COMPLETE : 0,
MEMTXATTRS_UNSPECIFIED);
- trace_igbvf_mig_dirty_query(s->vfn, (uint64_t)out_size, dirty_pages);
+ trace_igbvf_mig_dirty_query(s->vfn, (uint64_t)out_size, dirty_pages,
+ stats->dma_writes);
}
static void igbvf_mig_dirty_ctrl(IgbVfState *s, uint32_t val)
@@ -1135,6 +1187,7 @@ void igbvf_mig_state_reset(IgbVfState *s)
ms->mig_dirty_range_size = 0;
ms->mig_dirty_buf_addr = 0;
ms->mig_dirty_status = IGB_MIG_DIRTY_STATUS_OK;
+ memset(&igbvf_get_core(s)->vf_mig_stats[s->vfn], 0, sizeof(IgbVfMigStats));
ms->mig_saved_vfre = true;
ms->mig_saved_vfte = true;
trace_igbvf_mig_reset(s->vfn);
diff --git a/hw/net/trace-events b/hw/net/trace-events
index 1e38d9ab3697..76584b43bf19 100644
--- a/hw/net/trace-events
+++ b/hw/net/trace-events
@@ -305,7 +305,7 @@ igbvf_mig_load_state(uint16_t vfn, uint32_t size, bool vfre, bool vfte) "VF%u: l
igbvf_mig_reset(uint16_t vfn) "VF%u: migration state reset"
igbvf_mig_dirty_enable(uint16_t vfn, uint64_t pgsize, uint64_t nbits) "VF%u: dirty tracking enabled pgsize=%"PRIu64" nbits=%"PRIu64
igbvf_mig_dirty_disable(uint16_t vfn) "VF%u: dirty tracking disabled"
-igbvf_mig_dirty_query(uint16_t vfn, uint64_t size, uint32_t dirty_pages) "VF%u: dirty query returned %"PRIu64" bytes, %u dirty pages"
+igbvf_mig_dirty_query(uint16_t vfn, uint64_t size, uint32_t dirty_pages, uint64_t dma_writes) "VF%u: dirty query returned %"PRIu64" bytes, %u dirty pages (dma_writes=%"PRIu64")"
# igb_core.c - VF migration diagnostics
igb_core_dirty_track_dma(int vfn, uint64_t addr, uint64_t len) "VF%d: dirty DMA addr=0x%"PRIx64" len=%"PRIu64
--
2.55.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* Re: [RFC PATCH 02/11] igb: Add x-vf-migration property and vendor-specific capability for IGBVF
2026-07-27 5:39 ` [RFC PATCH 02/11] igb: Add x-vf-migration property and vendor-specific capability for IGBVF Cédric Le Goater
@ 2026-07-27 7:13 ` Akihiko Odaki
0 siblings, 0 replies; 26+ messages in thread
From: Akihiko Odaki @ 2026-07-27 7:13 UTC (permalink / raw)
To: Cédric Le Goater, qemu-devel
Cc: Sriram Yagnaraman, Jason Wang, Alex Williamson,
Michael S . Tsirkin, Peter Xu, Avihai Horon
The patch subject is a bit long.
On 2026/07/27 14:39, Cédric Le Goater wrote:
> Add a PF-level "x-vf-migration" boolean property (default off) that,
> when enabled, causes each emulated VF to advertise a vendor-specific
> PCI capability in its config space. The variant driver igb-vfio-pci
> probes for this capability at bind time and uses its presence to
> enable migration support.
>
> The capability is a 16-byte vendor-specific capability (PCI_CAP_ID_VNDR)
> containing a magic value ("MIGB"), the BAR index where the migration
> region will be mapped, and feature flags indicating which migration
> features are supported.
>
> Assisted-by: Claude
> Suggested-by: Alex Williamson <alex@shazbot.org>
> Signed-off-by: Cédric Le Goater <clg@redhat.com>
> ---
> MAINTAINERS | 6 +++
> docs/system/device-emulation.rst | 1 +
> docs/system/devices/igb-migration.rst | 20 ++++++++
> docs/system/devices/igb.rst | 6 +++
> hw/net/igb_migration.h | 43 ++++++++++++++++
> hw/net/igb.c | 12 +++++
> hw/net/igb_migration.c | 72 +++++++++++++++++++++++++++
> hw/net/igbvf.c | 11 ++++
> hw/net/meson.build | 2 +-
> hw/net/trace-events | 1 +
> 10 files changed, 173 insertions(+), 1 deletion(-)
> create mode 100644 docs/system/devices/igb-migration.rst
> create mode 100644 hw/net/igb_migration.h
> create mode 100644 hw/net/igb_migration.c
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index a28935c89866..6d29b905c47c 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -2765,6 +2765,12 @@ F: tests/functional/x86_64/test_netdev_ethtool.py
> F: tests/qtest/igb-test.c
> F: tests/qtest/libqos/igb.c
>
> +igb VF migration
> +M: Cédric Le Goater <clg@redhat.com>
> +S: Maintained
> +F: hw/net/igb_migration.*
> +F: docs/system/devices/igb-migration.rst
> +
> eepro100
> M: Stefan Weil <sw@weilnetz.de>
> S: Maintained
> diff --git a/docs/system/device-emulation.rst b/docs/system/device-emulation.rst
> index 40054bb7dfcc..75f423b795cd 100644
> --- a/docs/system/device-emulation.rst
> +++ b/docs/system/device-emulation.rst
> @@ -90,6 +90,7 @@ Emulated Devices
> devices/cxl.rst
> devices/emmc.rst
> devices/igb.rst
> + devices/igb-migration.rst
> devices/ivshmem-flat.rst
> devices/ivshmem.rst
> devices/keyboard.rst
> diff --git a/docs/system/devices/igb-migration.rst b/docs/system/devices/igb-migration.rst
> new file mode 100644
> index 000000000000..017b47c4544e
> --- /dev/null
> +++ b/docs/system/devices/igb-migration.rst
> @@ -0,0 +1,20 @@
> +.. SPDX-License-Identifier: GPL-2.0-or-later
> +.. _igb-migration:
> +
> +igb VF Migration
> +----------------
> +
> +The igb device supports an experimental VF migration interface that allows
> +the ``igb-vfio-pci`` variant driver to migrate VF state during live
> +migration. This is enabled with the ``x-vf-migration`` property::
> +
> + -device igb,x-vf-migration=on,...
> +
> +When enabled, each emulated VF advertises a vendor-specific PCI capability
> +(cap id 0x09) with a magic signature (``0x4D494742`` / "MIGB") that the
> +variant driver probes at bind time. The capability contains an interface
> +version number, the BAR index hosting the migration register region, and
> +feature flags indicating which migration features are supported.
> +
> +This feature is experimental and the ``x-`` prefix indicates the interface
> +may change.
> diff --git a/docs/system/devices/igb.rst b/docs/system/devices/igb.rst
> index 50f625fd77e4..00271dbc92c3 100644
> --- a/docs/system/devices/igb.rst
> +++ b/docs/system/devices/igb.rst
> @@ -64,6 +64,12 @@ command:
>
> pyvenv/bin/meson test --suite thorough func-x86_64-netdev_ethtool
>
> +VF Migration (experimental)
> +===========================
> +
> +See :ref:`igb-migration` for details on the experimental VF live migration
> +interface.
> +
> References
> ==========
>
> diff --git a/hw/net/igb_migration.h b/hw/net/igb_migration.h
> new file mode 100644
> index 000000000000..e79892436b21
> --- /dev/null
> +++ b/hw/net/igb_migration.h
> @@ -0,0 +1,43 @@
> +/*
> + * QEMU Intel 82576 SR/IOV VF Migration Support
> + *
> + * Copyright (c) 2026 Red Hat, Inc.
> + *
> + * SPDX-License-Identifier: GPL-2.0-or-later
> + */
> +
> +#ifndef IGB_MIGRATION_H
> +#define IGB_MIGRATION_H
> +
> +#include "hw/pci/pci_device.h"
> +
> +/* Migration BAR definitions */
> +#define IGB_MIG_BAR_IDX (2)
> +#define IGB_MIG_BAR_SIZE (64 * 1024)
> +
> +/*
> + * Vendor-specific PCI capability for migration discovery.
> + *
> + * The igb-vfio-pci variant driver probes for this at bind time.
> + * If present, the driver knows the emulated VF supports migration.
> + */
> +#define IGB_MIG_CAP_MAGIC 0x4D494742 /* "MIGB" */
> +#define IGB_MIG_CAP_VERSION 1
> +
> +#define IGB_MIG_CAP_F_STATE (1u << 0) /* device state serialization */
> +#define IGB_MIG_CAP_F_DIRTY (1u << 1) /* dirty page tracking */
> +
> +#define IGB_MIG_CAP_SIZE 16
> +#define IGB_MIG_CAP_OFF_MAGIC 4 /* offset within cap for magic field */
> +#define IGB_MIG_CAP_OFF_BARID 8 /* offset within cap for BAR id */
> +#define IGB_MIG_CAP_OFF_FLAGS 12 /* offset within cap for feature flags */
> +
> +typedef struct IgbVfMigState {
> + bool migration_cap;
> + MemoryRegion mig_bar;
> +} IgbVfMigState;
> +
> +void igb_pf_init_migration_bar(PCIDevice *dev);
> +bool igbvf_add_migration_cap(PCIDevice *dev, Error **errp);
> +
> +#endif
> diff --git a/hw/net/igb.c b/hw/net/igb.c
> index c076807e7110..b43235996db2 100644
> --- a/hw/net/igb.c
> +++ b/hw/net/igb.c
> @@ -57,6 +57,7 @@
>
> #include "igb_common.h"
> #include "igb_core.h"
> +#include "igb_migration.h"
>
> #include "trace.h"
> #include "qapi/error.h"
> @@ -69,6 +70,7 @@ struct IGBState {
> PCIDevice parent_obj;
> NICState *nic;
> NICConf conf;
> + bool vf_migration;
>
> MemoryRegion mmio;
> MemoryRegion flash;
> @@ -459,6 +461,15 @@ static void igb_pci_realize(PCIDevice *pci_dev, Error **errp)
> pcie_sriov_pf_init_vf_bar(pci_dev, IGBVF_MSIX_BAR_IDX,
> PCI_BASE_ADDRESS_MEM_TYPE_64 | PCI_BASE_ADDRESS_MEM_PREFETCH,
> IGBVF_MSIX_SIZE);
> + /*
> + * When VF migration support is enabled, register an additional VF
> + * BAR for the migration register region. The variant driver
> + * discovers this via a vendor-specific PCI capability that points
> + * to this BAR.
> + */
> + if (s->vf_migration) {
> + igb_pf_init_migration_bar(pci_dev);
> + }
>
> igb_init_net_peer(s, pci_dev, macaddr);
>
> @@ -597,6 +608,7 @@ static const VMStateDescription igb_vmstate = {
> static const Property igb_properties[] = {
> DEFINE_NIC_PROPERTIES(IGBState, conf),
> DEFINE_PROP_BOOL("x-pcie-flr-init", IGBState, has_flr, true),
> + DEFINE_PROP_BOOL("x-vf-migration", IGBState, vf_migration, false),
> };
>
> static void igb_class_init(ObjectClass *class, const void *data)
> diff --git a/hw/net/igb_migration.c b/hw/net/igb_migration.c
> new file mode 100644
> index 000000000000..794f1217d3e5
> --- /dev/null
> +++ b/hw/net/igb_migration.c
> @@ -0,0 +1,72 @@
> +/*
> + * QEMU Intel 82576 SR/IOV VF Migration Support
> + *
> + * Copyright (c) 2026 Red Hat, Inc.
> + *
> + * SPDX-License-Identifier: GPL-2.0-or-later
> + */
> +
> +#include "qemu/osdep.h"
> +#include "hw/pci/pci_device.h"
> +#include "hw/pci/pcie.h"
> +#include "igb_common.h"
> +#include "igb_migration.h"
> +#include "trace.h"
> +
> +
> +
> +/*
> + * 32-bit prefetchable BAR. A 64-bit BAR2 would consume BAR2+BAR3, but
> + * BAR3 is already used for MSI-X (IGBVF_MSIX_BAR_IDX = 3).
> + *
> + * BAR Index Type Size Purpose
> + * BAR0 0+1 64-bit prefetchable 16 KB MMIO registers
> + * BAR2 2 32-bit prefetchable 64 KB Migration
> + * BAR3 3+4 64-bit prefetchable 16 KB MSI-X table + PBA
> + * BAR5 5 - - Unused
> + */
> +void igb_pf_init_migration_bar(PCIDevice *dev)
> +{
> + pcie_sriov_pf_init_vf_bar(dev, IGB_MIG_BAR_IDX,
> + PCI_BASE_ADDRESS_MEM_PREFETCH,
> + IGB_MIG_BAR_SIZE);
> +}
> +
> +/*
> + * Add vendor-specific PCI capability that the variant driver probes for.
> + *
> + * Layout (16 bytes):
> + * [0] cap_id (PCI_CAP_ID_VNDR = 0x09)
> + * [1] next_cap
> + * [2] cap_len (16)
> + * [3] version (IGB_MIG_CAP_VERSION)
> + * [4-7] magic (IGB_MIG_CAP_MAGIC, little-endian)
> + * [8-11] bar_id (IGB_MIG_BAR_IDX, little-endian)
> + * [12-15] flags (feature flags, little-endian)
> + */
> +bool igbvf_add_migration_cap(PCIDevice *dev, Error **errp)
> +{
> + int offset;
> +
> + offset = pci_add_capability(dev, PCI_CAP_ID_VNDR, 0,
> + IGB_MIG_CAP_SIZE, errp);
> + if (offset < 0) {
> + return false;
> + }
> +
> + /* Length and version in the standard cap flags word */
> + pci_set_byte(dev->config + offset + PCI_CAP_FLAGS,
> + IGB_MIG_CAP_SIZE);
> + pci_set_byte(dev->config + offset + PCI_CAP_FLAGS + 1,
> + IGB_MIG_CAP_VERSION);
> +
> + pci_set_long(dev->config + offset + IGB_MIG_CAP_OFF_MAGIC,
> + IGB_MIG_CAP_MAGIC);
> + pci_set_long(dev->config + offset + IGB_MIG_CAP_OFF_BARID,
> + IGB_MIG_BAR_IDX);
> + pci_set_long(dev->config + offset + IGB_MIG_CAP_OFF_FLAGS,
> + IGB_MIG_CAP_F_STATE);
It advertises IGB_MIG_CAP_F_STATE before registering BAR2’s
MemoryRegion. The companion driver therefore discovers migration support
and tries to use an unimplemented BAR. Please change the patch series to
publish the capability with the BAR implementation or squash the commits.
Regards,
Akihiko Odaki
> +
> + trace_igbvf_mig_cap_add(pcie_sriov_vf_number(dev), offset);
> + return true;
> +}
> diff --git a/hw/net/igbvf.c b/hw/net/igbvf.c
> index 9a165c7063ee..94c9739cd58c 100644
> --- a/hw/net/igbvf.c
> +++ b/hw/net/igbvf.c
> @@ -47,6 +47,7 @@
> #include "net/net.h"
> #include "igb_common.h"
> #include "igb_core.h"
> +#include "igb_migration.h"
> #include "trace.h"
> #include "qapi/error.h"
>
> @@ -57,6 +58,8 @@ struct IgbVfState {
>
> MemoryRegion mmio;
> MemoryRegion msix;
> +
> + IgbVfMigState mig;
> };
>
> static hwaddr vf_to_pf_addr(hwaddr addr, uint16_t vfn, bool write)
> @@ -272,6 +275,14 @@ static void igbvf_pci_realize(PCIDevice *dev, Error **errp)
> hw_error("Failed to initialize PCIe capability");
> }
>
> + s->mig.migration_cap = object_property_get_bool(OBJECT(pcie_sriov_get_pf(dev)),
> + "x-vf-migration", &error_abort);
> + if (s->mig.migration_cap) {
> + if (!igbvf_add_migration_cap(dev, errp)) {
> + return;
> + }
> + }
> +
> if (object_property_get_bool(OBJECT(pcie_sriov_get_pf(dev)),
> "x-pcie-flr-init", &error_abort)) {
> pcie_cap_flr_init(dev);
> diff --git a/hw/net/meson.build b/hw/net/meson.build
> index 84f142df222a..bb4b449b25ba 100644
> --- a/hw/net/meson.build
> +++ b/hw/net/meson.build
> @@ -11,7 +11,7 @@ system_ss.add(when: 'CONFIG_E1000_PCI', if_true: files('e1000.c', 'e1000x_common
> system_ss.add(when: 'CONFIG_E1000E_PCI_EXPRESS', if_true: files('net_tx_pkt.c', 'net_rx_pkt.c'))
> system_ss.add(when: 'CONFIG_E1000E_PCI_EXPRESS', if_true: files('e1000e.c', 'e1000e_core.c', 'e1000x_common.c'))
> system_ss.add(when: 'CONFIG_IGB_PCI_EXPRESS', if_true: files('net_tx_pkt.c', 'net_rx_pkt.c'))
> -system_ss.add(when: 'CONFIG_IGB_PCI_EXPRESS', if_true: files('igb.c', 'igbvf.c', 'igb_core.c'))
> +system_ss.add(when: 'CONFIG_IGB_PCI_EXPRESS', if_true: files('igb.c', 'igbvf.c', 'igb_core.c', 'igb_migration.c'))
> system_ss.add(when: 'CONFIG_RTL8139_PCI', if_true: files('rtl8139.c'))
> system_ss.add(when: 'CONFIG_TULIP', if_true: files('tulip.c'))
> system_ss.add(when: 'CONFIG_VMXNET3_PCI', if_true: files('net_tx_pkt.c', 'net_rx_pkt.c'))
> diff --git a/hw/net/trace-events b/hw/net/trace-events
> index 001a20b0e2ac..06d8848023e5 100644
> --- a/hw/net/trace-events
> +++ b/hw/net/trace-events
> @@ -294,6 +294,7 @@ igb_wrn_rx_desc_modes_not_supp(int desc_type) "Not supported descriptor type: %d
>
> # igbvf.c
> igbvf_wrn_io_addr_unknown(uint64_t addr) "IO unknown register 0x%"PRIx64
> +igbvf_mig_cap_add(uint16_t vfn, int offset) "VF%u: added migration vendor cap at config offset 0x%x"
>
> # spapr_llan.c
> spapr_vlan_get_rx_bd_from_pool_found(int pool, int32_t count, uint32_t rx_bufs) "pool=%d count=%"PRId32" rxbufs=%"PRIu32
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH 03/11] igb: Add migration BAR with state machine
2026-07-27 5:39 ` [RFC PATCH 03/11] igb: Add migration BAR with state machine Cédric Le Goater
@ 2026-07-27 7:16 ` Akihiko Odaki
0 siblings, 0 replies; 26+ messages in thread
From: Akihiko Odaki @ 2026-07-27 7:16 UTC (permalink / raw)
To: Cédric Le Goater, qemu-devel
Cc: Sriram Yagnaraman, Jason Wang, Alex Williamson,
Michael S . Tsirkin, Peter Xu, Avihai Horon
On 2026/07/27 14:39, Cédric Le Goater wrote:
> When x-vf-migration=on, register a 64KB migration BAR (BAR2) on each
> emulated VF. This BAR implements a VFIO-like migration state machine
> that the igb-vfio-pci variant driver uses to serialize/deserialize VF
> device state during live migration.
>
> The migration register region is laid out as:
>
> 0x000 DEVICE_STATE (RW) - migration state machine control
> 0x004 MIG_STATUS (RO) - status flags (DATA_AVAIL, ERROR,
> QUIESCED) + error code in [15:8]
> 0x008 MIG_CAPS (RO) - advertised capabilities
> 0x00C MIG_VERSION (RO) - interface version
> 0x010 DATA_SIZE (RW) - max state size at reset, actual after save
> 0x014 DATA_XFER (WO) - trigger DMA save or DMA load
> 0x018 DATA_BUF_ADDR_LO (WO) - low 32 bits of state DMA buffer
> 0x01C DATA_BUF_ADDR_HI (WO) - high 32 bits of state DMA buffer
>
> State data is transferred via a driver-provided DMA buffer. The driver
> writes its PF DMA address to DATA_BUF_ADDR_LO/HI and triggers the
> transfer with DATA_XFER. The device DMA-writes the serialized state on
> save and DMA-reads it on restore. DMA is performed through the PF
> device (pcie_sriov_get_pf) because VFIO owns the VF's IOMMU domain.
>
> VF state serialization is added in the next patch.
>
> Assisted-by: Claude
> Suggested-by: Alex Williamson <alex@shazbot.org>
> Signed-off-by: Cédric Le Goater <clg@redhat.com>
> ---
> docs/system/devices/igb-migration.rst | 43 ++++
> hw/net/igb_common.h | 11 +
> hw/net/igb_migration.h | 52 +++++
> hw/net/igb_migration.c | 293 ++++++++++++++++++++++++++
> hw/net/igbvf.c | 18 +-
> hw/net/trace-events | 8 +
> 6 files changed, 416 insertions(+), 9 deletions(-)
>
> diff --git a/docs/system/devices/igb-migration.rst b/docs/system/devices/igb-migration.rst
> index 017b47c4544e..04c44ef072af 100644
> --- a/docs/system/devices/igb-migration.rst
> +++ b/docs/system/devices/igb-migration.rst
> @@ -18,3 +18,46 @@ feature flags indicating which migration features are supported.
>
> This feature is experimental and the ``x-`` prefix indicates the interface
> may change.
> +
> +Migration BAR layout
> +~~~~~~~~~~~~~~~~~~~~
> +
> +The migration BAR (BAR2, 64 KB) implements a VFIO-like state machine with
> +the following register layout::
> +
> + Offset Name Access Description
> + 0x000 DEVICE_STATE RW Migration state (RUNNING=2, STOP=1,
> + STOP_COPY=3, RESUMING=4, PRE_COPY=5)
> + 0x004 STATUS RO Flags[2:0]: DATA_AVAIL, ERROR, QUIESCED
> + Error code[15:8] (when ERROR is set)
> + 0x008 CAPS RO F_STATE, F_DIRTY, max_ranges[11:8],
> + pgsizes[31:12]
> + 0x00C VERSION RO Interface version (1)
> + 0x010 DATA_SIZE RW Max state size at reset, actual after save
> + 0x014 DATA_XFER WO Trigger DMA save or DMA load
> + 0x018 DATA_BUF_ADDR_LO WO Low 32 bits of state DMA buffer address
> + 0x01C DATA_BUF_ADDR_HI WO High 32 bits of state DMA buffer address
> +
> +State transitions follow the VFIO migration state machine: the driver
> +writes to ``DEVICE_STATE`` to move between states and reads ``STATUS``
> +to check for completion.
> +
> +State data is transferred via a driver-provided DMA buffer. The driver
> +writes its PF DMA address to ``DATA_BUF_ADDR_LO/HI`` and triggers the
> +transfer with ``DATA_XFER``. The device DMA-writes the serialized state
> +on save and DMA-reads it on restore. DMA is performed through the PF
> +device because VFIO owns the VF's IOMMU domain.
> +
> +The state blob is a versioned sequence of register (offset, value)
> +pairs.
> +
> +When ``STATUS`` has the ``ERROR`` bit set, bits [15:8] contain an error
> +code identifying the failure::
> +
> + 0 (none) No error
> + 1 BAD_MAGIC State blob magic mismatch
> + 2 BAD_VERSION State blob version mismatch
> + 3 BAD_SIZE State blob too large or empty
> + 4 BAD_VFN VF number mismatch (source != destination)
> + 5 DMA_FAILED DMA transfer to/from state buffer failed
> + 6 NO_BUFFER DATA_XFER without buffer address set
> diff --git a/hw/net/igb_common.h b/hw/net/igb_common.h
> index b316a5bcfa5c..01816e002c23 100644
> --- a/hw/net/igb_common.h
> +++ b/hw/net/igb_common.h
> @@ -27,6 +27,7 @@
> #define HW_NET_IGB_COMMON_H
>
> #include "igb_regs.h"
> +#include "igb_migration.h"
>
> #define TYPE_IGBVF "igbvf"
>
> @@ -36,6 +37,16 @@
> #define IGBVF_MMIO_SIZE (16 * 1024)
> #define IGBVF_MSIX_SIZE (16 * 1024)
>
> +struct IgbVfState {
> + PCIDevice parent_obj;
> + uint16_t vfn;
> +
> + MemoryRegion mmio;
> + MemoryRegion msix;
> +
> + IgbVfMigState mig;
> +};
> +
> #define defreg(x) x = (E1000_##x >> 2)
> #define defreg_indexed(x, i) x##i = (E1000_##x(i) >> 2)
> #define defreg_indexeda(x, i) x##i##_A = (E1000_##x##_A(i) >> 2)
> diff --git a/hw/net/igb_migration.h b/hw/net/igb_migration.h
> index e79892436b21..739a189810b0 100644
> --- a/hw/net/igb_migration.h
> +++ b/hw/net/igb_migration.h
> @@ -32,12 +32,64 @@
> #define IGB_MIG_CAP_OFF_BARID 8 /* offset within cap for BAR id */
> #define IGB_MIG_CAP_OFF_FLAGS 12 /* offset within cap for feature flags */
>
> +/*
> + * Maximum serialized VF state size, sized to hold all per-VF
> + * registers plus TX context descriptors with room to spare.
> + */
> +#define IGB_VF_STATE_MAX_SIZE 4096
> +
> +/*
> + * Migration BAR register offsets.
> + */
> +#define IGB_MIG_DEVICE_STATE 0x000
> +#define IGB_MIG_STATUS 0x004
> +#define IGB_MIG_CAPS 0x008
> +#define IGB_MIG_VERSION 0x00C
> +#define IGB_MIG_DATA_SIZE 0x010
> +#define IGB_MIG_DATA_XFER 0x014
> +#define IGB_MIG_DATA_BUF_ADDR_LO 0x018
> +#define IGB_MIG_DATA_BUF_ADDR_HI 0x01C
> +
> +/* DEVICE_STATE values - mirrors VFIO migration states */
> +#define IGB_MIG_STATE_ERROR 0
> +#define IGB_MIG_STATE_STOP 1
> +#define IGB_MIG_STATE_RUNNING 2
> +#define IGB_MIG_STATE_STOP_COPY 3
> +#define IGB_MIG_STATE_RESUMING 4
> +#define IGB_MIG_STATE_PRE_COPY 5
> +
> +/* MIG_STATUS bits */
> +#define IGB_MIG_STATUS_DATA_AVAIL (1u << 0)
> +#define IGB_MIG_STATUS_ERROR (1u << 1)
> +
> +/* MIG_STATUS error codes in bits [15:8], valid when ERROR bit is set */
> +#define IGB_MIG_STATUS_ERR_SHIFT 8
> +#define IGB_MIG_STATUS_ERR_MASK (0xffu << IGB_MIG_STATUS_ERR_SHIFT)
> +#define IGB_MIG_STATUS_ERR(code) (IGB_MIG_STATUS_ERROR | \
> + ((uint32_t)(code) << IGB_MIG_STATUS_ERR_SHIFT))
> +
> +#define IGB_MIG_ERR_BAD_MAGIC 1
> +#define IGB_MIG_ERR_BAD_VERSION 2
> +#define IGB_MIG_ERR_BAD_SIZE 3
> +#define IGB_MIG_ERR_BAD_VFN 4
> +#define IGB_MIG_ERR_DMA_FAILED 5
> +#define IGB_MIG_ERR_NO_BUFFER 6
> +
> typedef struct IgbVfMigState {
> bool migration_cap;
> MemoryRegion mig_bar;
> +
> + uint32_t mig_state;
> + uint8_t mig_error;
> + uint8_t mig_data[IGB_VF_STATE_MAX_SIZE];
> + uint32_t mig_data_size;
> + uint64_t mig_data_buf_addr;
> } IgbVfMigState;
>
> +typedef struct IgbVfState IgbVfState;
> void igb_pf_init_migration_bar(PCIDevice *dev);
> bool igbvf_add_migration_cap(PCIDevice *dev, Error **errp);
> +void igbvf_mig_bar_init(IgbVfState *s);
> +void igbvf_mig_state_reset(IgbVfState *s);
>
> #endif
> diff --git a/hw/net/igb_migration.c b/hw/net/igb_migration.c
> index 794f1217d3e5..a0d044815fd9 100644
> --- a/hw/net/igb_migration.c
> +++ b/hw/net/igb_migration.c
> @@ -7,9 +7,13 @@
> */
>
> #include "qemu/osdep.h"
> +#include "qemu/log.h"
> #include "hw/pci/pci_device.h"
> #include "hw/pci/pcie.h"
> +#include "net/eth.h"
> +#include "net/net.h"
> #include "igb_common.h"
> +#include "igb_core.h"
> #include "igb_migration.h"
> #include "trace.h"
>
> @@ -70,3 +74,292 @@ bool igbvf_add_migration_cap(PCIDevice *dev, Error **errp)
> trace_igbvf_mig_cap_add(pcie_sriov_vf_number(dev), offset);
> return true;
> }
> +
> +/*
> + * =====================================================================
> + * Per-VF state serialization / deserialization
> + * =====================================================================
> + */
> +
> +static int igb_core_vf_save_state(IgbVfState *s,
> + void *buf, size_t buf_size)
> +{
> + int size = 0;
> +
> + trace_igbvf_mig_save_state(s->vfn, size);
> + return size;
> +}
> +
> +static int igb_core_vf_max_data_size(IgbVfState *s)
> +{
> + int size = igb_core_vf_save_state(s, NULL, 0);
> +
> + g_assert(size > 0 && size <= IGB_VF_STATE_MAX_SIZE);
Enabling x-vf-migration aborts QEMU during realization: the state-save
stub returns zero and igb_core_vf_max_data_size() asserts that the
result is positive. Patch 4 happens to repair this, so patch 3 is not
independently usable.
Regards,
Akihiko Odaki
> + return size;
> +}
> +
> +static int igb_core_vf_load_state(IgbVfState *s,
> + const void *buf, size_t size)
> +{
> + trace_igbvf_mig_load_state(s->vfn, (uint32_t)size);
> + return 0;
> +}
> +
> +static int igbvf_mig_load(IgbVfState *s, const void *buf, size_t size)
> +{
> + int ret;
> +
> + ret = igb_core_vf_load_state(s, buf, size);
> + if (ret < 0) {
> + return ret;
> + }
> +
> + return 0;
> +}
> +
> +/* ================================================================
> + * Migration BAR register read/write handlers
> + * ================================================================ */
> +
> +static bool igbvf_mig_set_state(IgbVfState *s, uint32_t new_state)
> +{
> + IgbVfMigState *ms = &s->mig;
> + uint32_t old = ms->mig_state;
> + int ret;
> +
> + switch (new_state) {
> + case IGB_MIG_STATE_STOP:
> + if (old != IGB_MIG_STATE_RUNNING &&
> + old != IGB_MIG_STATE_STOP_COPY &&
> + old != IGB_MIG_STATE_RESUMING &&
> + old != IGB_MIG_STATE_ERROR) {
> + return false;
> + }
> + /* Restore DATA_SIZE to max, same as at reset */
> + ms->mig_data_size = igb_core_vf_max_data_size(s);
> + break;
> +
> + case IGB_MIG_STATE_RUNNING:
> + if (old != IGB_MIG_STATE_STOP) {
> + return false;
> + }
> + break;
> +
> + case IGB_MIG_STATE_STOP_COPY:
> + if (old != IGB_MIG_STATE_STOP) {
> + return false;
> + }
> + ret = igb_core_vf_save_state(s, ms->mig_data, sizeof(ms->mig_data));
> + if (ret < 0) {
> + ms->mig_error = -ret;
> + ms->mig_state = IGB_MIG_STATE_ERROR;
> + return false;
> + }
> + ms->mig_data_size = ret;
> + break;
> +
> + case IGB_MIG_STATE_RESUMING:
> + if (old != IGB_MIG_STATE_STOP) {
> + return false;
> + }
> + memset(ms->mig_data, 0, sizeof(ms->mig_data));
> + ms->mig_data_size = 0;
> + break;
> +
> + default:
> + trace_igbvf_mig_set_state_err(s->vfn, old, new_state);
> + return false;
> + }
> +
> + ms->mig_state = new_state;
> + trace_igbvf_mig_set_state(s->vfn, old, new_state);
> + return true;
> +}
> +
> +static uint32_t igbvf_mig_get_status(IgbVfState *s)
> +{
> + IgbVfMigState *ms = &s->mig;
> + uint32_t status = 0;
> +
> + if (ms->mig_state == IGB_MIG_STATE_ERROR) {
> + status |= IGB_MIG_STATUS_ERR(ms->mig_error);
> + }
> + if (ms->mig_state == IGB_MIG_STATE_STOP_COPY && ms->mig_data_size > 0) {
> + status |= IGB_MIG_STATUS_DATA_AVAIL;
> + }
> +
> + return status;
> +}
> +
> +static void igbvf_mig_data_xfer(IgbVfState *s, uint32_t val)
> +{
> + IgbVfMigState *ms = &s->mig;
> + MemTxResult r;
> + int ret;
> +
> + if (!ms->mig_data_buf_addr) {
> + ms->mig_error = IGB_MIG_ERR_NO_BUFFER;
> + ms->mig_state = IGB_MIG_STATE_ERROR;
> + return;
> + }
> +
> + switch (ms->mig_state) {
> + case IGB_MIG_STATE_STOP_COPY:
> + /* Save: DMA-write serialized state to driver buffer */
> + r = pci_dma_write(pcie_sriov_get_pf(PCI_DEVICE(s)),
> + ms->mig_data_buf_addr,
> + ms->mig_data, ms->mig_data_size);
> + if (r != MEMTX_OK) {
> + qemu_log_mask(LOG_GUEST_ERROR,
> + "igbvf: VF%u state write failed at 0x%" PRIx64 "\n",
> + s->vfn, ms->mig_data_buf_addr);
> + ms->mig_error = IGB_MIG_ERR_DMA_FAILED;
> + ms->mig_state = IGB_MIG_STATE_ERROR;
> + }
> + break;
> +
> + case IGB_MIG_STATE_RESUMING:
> + /* Restore: DMA-read state from driver buffer and deserialize */
> + if (ms->mig_data_size == 0 ||
> + ms->mig_data_size > sizeof(ms->mig_data)) {
> + ms->mig_error = IGB_MIG_ERR_BAD_SIZE;
> + ms->mig_state = IGB_MIG_STATE_ERROR;
> + break;
> + }
> +
> + r = pci_dma_read(pcie_sriov_get_pf(PCI_DEVICE(s)),
> + ms->mig_data_buf_addr,
> + ms->mig_data, ms->mig_data_size);
> + if (r != MEMTX_OK) {
> + qemu_log_mask(LOG_GUEST_ERROR,
> + "igbvf: VF%u state read failed at 0x%" PRIx64 "\n",
> + s->vfn, ms->mig_data_buf_addr);
> + ms->mig_error = IGB_MIG_ERR_DMA_FAILED;
> + ms->mig_state = IGB_MIG_STATE_ERROR;
> + break;
> + }
> +
> + ret = igbvf_mig_load(s, ms->mig_data, ms->mig_data_size);
> + if (ret < 0) {
> + ms->mig_error = -ret;
> + ms->mig_state = IGB_MIG_STATE_ERROR;
> + }
> + break;
> +
> + default:
> + break;
> + }
> +}
> +
> +static uint64_t igbvf_mig_read(void *opaque, hwaddr addr, unsigned size)
> +{
> + IgbVfState *s = opaque;
> + IgbVfMigState *ms = &s->mig;
> + uint64_t val = 0;
> +
> + switch (addr) {
> + case IGB_MIG_DEVICE_STATE:
> + val = ms->mig_state;
> + break;
> + case IGB_MIG_STATUS:
> + val = igbvf_mig_get_status(s);
> + break;
> + case IGB_MIG_CAPS:
> + val = IGB_MIG_CAP_F_STATE;
> + break;
> + case IGB_MIG_VERSION:
> + val = IGB_MIG_CAP_VERSION;
> + break;
> + case IGB_MIG_DATA_SIZE:
> + val = ms->mig_data_size;
> + break;
> + default:
> + qemu_log_mask(LOG_GUEST_ERROR,
> + "igbvf: VF%u bad migration BAR read at 0x%"
> + HWADDR_PRIx "\n", s->vfn, addr);
> + break;
> + }
> +
> + trace_igbvf_mig_bar_read(s->vfn, addr, val);
> +
> + return val;
> +}
> +
> +static void igbvf_mig_write(void *opaque, hwaddr addr, uint64_t val,
> + unsigned size)
> +{
> + IgbVfState *s = opaque;
> + IgbVfMigState *ms = &s->mig;
> +
> + trace_igbvf_mig_bar_write(s->vfn, addr, val);
> +
> + switch (addr) {
> + case IGB_MIG_DEVICE_STATE:
> + igbvf_mig_set_state(s, (uint32_t)val);
> + break;
> + case IGB_MIG_DATA_SIZE:
> + if (val <= sizeof(ms->mig_data)) {
> + ms->mig_data_size = (uint32_t)val;
> + } else {
> + qemu_log_mask(LOG_GUEST_ERROR,
> + "igbvf: VF%u DATA_SIZE %" PRIu64 " exceeds max %zu\n",
> + s->vfn, val, sizeof(ms->mig_data));
> + }
> + break;
> + case IGB_MIG_DATA_XFER:
> + igbvf_mig_data_xfer(s, (uint32_t)val);
> + break;
> + case IGB_MIG_DATA_BUF_ADDR_LO:
> + ms->mig_data_buf_addr =
> + deposit64(ms->mig_data_buf_addr, 0, 32, val);
> + break;
> + case IGB_MIG_DATA_BUF_ADDR_HI:
> + ms->mig_data_buf_addr =
> + deposit64(ms->mig_data_buf_addr, 32, 32, val);
> + break;
> + default:
> + qemu_log_mask(LOG_GUEST_ERROR,
> + "igbvf: VF%u bad migration BAR write at 0x%"
> + HWADDR_PRIx "\n", s->vfn, addr);
> + break;
> + }
> +}
> +
> +static const MemoryRegionOps mig_bar_ops = {
> + .read = igbvf_mig_read,
> + .write = igbvf_mig_write,
> + .endianness = DEVICE_LITTLE_ENDIAN,
> + .impl = {
> + .min_access_size = 4,
> + .max_access_size = 4,
> + },
> +};
> +
> +/*
> + * Use the QEM-internal PCI_BASE_ADDRESS_MEM_ALWAYS_ON BAR type flag
> + * to keep the memory BAR always mapped.
> + */
> +void igbvf_mig_bar_init(IgbVfState *s)
> +{
> + IgbVfMigState *ms = &s->mig;
> +
> + memory_region_init_io(&ms->mig_bar, OBJECT(s), &mig_bar_ops, s,
> + "igbvf-mig", IGB_MIG_BAR_SIZE);
> + pci_register_bar(PCI_DEVICE(s), IGB_MIG_BAR_IDX,
> + PCI_BASE_ADDRESS_MEM_PREFETCH |
> + PCI_BASE_ADDRESS_MEM_ALWAYS_ON,
> + &ms->mig_bar);
> + trace_igbvf_mig_bar_init(s->vfn);
> +}
> +
> +void igbvf_mig_state_reset(IgbVfState *s)
> +{
> + IgbVfMigState *ms = &s->mig;
> +
> + ms->mig_state = IGB_MIG_STATE_RUNNING;
> + ms->mig_error = 0;
> + ms->mig_data_size = igb_core_vf_max_data_size(s);
> + ms->mig_data_buf_addr = 0;
> + memset(ms->mig_data, 0, sizeof(ms->mig_data));
> + trace_igbvf_mig_reset(s->vfn);
> +}
> diff --git a/hw/net/igbvf.c b/hw/net/igbvf.c
> index 94c9739cd58c..e9f9fc3369d8 100644
> --- a/hw/net/igbvf.c
> +++ b/hw/net/igbvf.c
> @@ -53,15 +53,6 @@
>
> OBJECT_DECLARE_SIMPLE_TYPE(IgbVfState, IGBVF)
>
> -struct IgbVfState {
> - PCIDevice parent_obj;
> -
> - MemoryRegion mmio;
> - MemoryRegion msix;
> -
> - IgbVfMigState mig;
> -};
> -
> static hwaddr vf_to_pf_addr(hwaddr addr, uint16_t vfn, bool write)
> {
> switch (addr) {
> @@ -281,6 +272,10 @@ static void igbvf_pci_realize(PCIDevice *dev, Error **errp)
> if (!igbvf_add_migration_cap(dev, errp)) {
> return;
> }
> +
> + s->vfn = pcie_sriov_vf_number(dev);
> + igbvf_mig_bar_init(s);
> + igbvf_mig_state_reset(s);
> }
>
> if (object_property_get_bool(OBJECT(pcie_sriov_get_pf(dev)),
> @@ -298,8 +293,13 @@ static void igbvf_pci_realize(PCIDevice *dev, Error **errp)
> static void igbvf_qdev_reset_hold(Object *obj, ResetType type)
> {
> PCIDevice *vf = PCI_DEVICE(obj);
> + IgbVfState *s = IGBVF(obj);
>
> igb_vf_reset(pcie_sriov_get_pf(vf), pcie_sriov_vf_number(vf));
> +
> + if (s->mig.migration_cap) {
> + igbvf_mig_state_reset(s);
> + }
> }
>
> static void igbvf_pci_uninit(PCIDevice *dev)
> diff --git a/hw/net/trace-events b/hw/net/trace-events
> index 06d8848023e5..0b13a99b3f32 100644
> --- a/hw/net/trace-events
> +++ b/hw/net/trace-events
> @@ -295,6 +295,14 @@ igb_wrn_rx_desc_modes_not_supp(int desc_type) "Not supported descriptor type: %d
> # igbvf.c
> igbvf_wrn_io_addr_unknown(uint64_t addr) "IO unknown register 0x%"PRIx64
> igbvf_mig_cap_add(uint16_t vfn, int offset) "VF%u: added migration vendor cap at config offset 0x%x"
> +igbvf_mig_bar_init(uint16_t vfn) "VF%u: migration BAR initialized"
> +igbvf_mig_bar_read(uint16_t vfn, uint64_t addr, uint64_t val) "VF%u: BAR read addr=0x%"PRIx64" val=0x%"PRIx64
> +igbvf_mig_bar_write(uint16_t vfn, uint64_t addr, uint64_t val) "VF%u: BAR write addr=0x%"PRIx64" val=0x%"PRIx64
> +igbvf_mig_set_state(uint16_t vfn, uint32_t old_state, uint32_t new_state) "VF%u: state %u -> %u"
> +igbvf_mig_set_state_err(uint16_t vfn, uint32_t old_state, uint32_t new_state) "VF%u: invalid transition %u -> %u"
> +igbvf_mig_save_state(uint16_t vfn, uint32_t size) "VF%u: saved %u bytes of device state"
> +igbvf_mig_load_state(uint16_t vfn, uint32_t size) "VF%u: loaded %u bytes of device state"
> +igbvf_mig_reset(uint16_t vfn) "VF%u: migration state reset"
>
> # spapr_llan.c
> spapr_vlan_get_rx_bd_from_pool_found(int pool, int32_t count, uint32_t rx_bufs) "pool=%d count=%"PRId32" rxbufs=%"PRIu32
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH 04/11] igb: Add VF state serialization for live migration
2026-07-27 5:39 ` [RFC PATCH 04/11] igb: Add VF state serialization for live migration Cédric Le Goater
@ 2026-07-27 7:31 ` Akihiko Odaki
0 siblings, 0 replies; 26+ messages in thread
From: Akihiko Odaki @ 2026-07-27 7:31 UTC (permalink / raw)
To: Cédric Le Goater, qemu-devel
Cc: Sriram Yagnaraman, Jason Wang, Alex Williamson,
Michael S . Tsirkin, Peter Xu, Avihai Horon
On 2026/07/27 14:39, Cédric Le Goater wrote:
> Add igb_pf_get_core() so migration code can reach the PF's IGBCore
> from a VF device and implement igb_core_vf_save_state() and
> igb_core_vf_load_state() to serialize and restore per-VF device state
> through the migration BAR.
>
> The wire format is a versioned blob: header (magic, version, VF
> number, register count), offset/value pairs for per-VF registers,
> dynamically scanned RA/RA2 entries, and TX context descriptors. PVT
> shadow registers (PVTEIMS/PVTEIAC/PVTEIAM) are saved instead of the PF
> aggregates which the L1 driver may have transiently cleared.
>
> The load path validates the header, restores registers to mac[], syncs
> EITR to eitr_guest_value[], and restores TX context. MSI-X table/PBA
> is not saved - L1's VFIO reprograms it after migration.
>
> Assisted-by: Claude
> Signed-off-by: Cédric Le Goater <clg@redhat.com>
> ---
> hw/net/igb_core.h | 1 +
> hw/net/igb.c | 6 +
> hw/net/igb_migration.c | 277 ++++++++++++++++++++++++++++++++++++++++-
> 3 files changed, 283 insertions(+), 1 deletion(-)
>
> diff --git a/hw/net/igb_core.h b/hw/net/igb_core.h
> index d70b54e318f1..58d4f57c99bb 100644
> --- a/hw/net/igb_core.h
> +++ b/hw/net/igb_core.h
> @@ -143,4 +143,5 @@ igb_receive_iov(IGBCore *core, const struct iovec *iov, int iovcnt);
> void
> igb_start_recv(IGBCore *core);
>
> +IGBCore *igb_pf_get_core(void *pf);
> #endif
> diff --git a/hw/net/igb.c b/hw/net/igb.c
> index b43235996db2..222413dd237a 100644
> --- a/hw/net/igb.c
> +++ b/hw/net/igb.c
> @@ -134,6 +134,12 @@ void igb_vf_reset(void *opaque, uint16_t vfn)
> igb_core_vf_reset(&s->core, vfn);
> }
>
> +IGBCore *igb_pf_get_core(void *pf)
> +{
> + IGBState *s = IGB(pf);
> + return &s->core;
> +}
> +
> static bool
> igb_io_get_reg_index(IGBState *s, uint32_t *idx)
> {
> diff --git a/hw/net/igb_migration.c b/hw/net/igb_migration.c
> index a0d044815fd9..61cf155a188d 100644
> --- a/hw/net/igb_migration.c
> +++ b/hw/net/igb_migration.c
> @@ -75,16 +75,226 @@ bool igbvf_add_migration_cap(PCIDevice *dev, Error **errp)
> return true;
> }
>
> +static IGBCore *igbvf_get_core(IgbVfState *s)
> +{
> + return igb_pf_get_core(pcie_sriov_get_pf(PCI_DEVICE(s)));
> +}
> +
> /*
> * =====================================================================
> * Per-VF state serialization / deserialization
> * =====================================================================
> + *
> + * Wire format:
> + * uint32_t magic (IGB_MIG_CAP_MAGIC)
> + * uint32_t version (1)
> + * uint32_t vfn (VF number)
> + * uint32_t num_regs (total register pairs, fixed + RA)
> + * { uint32_t offset; uint32_t value; } regs[num_regs]
> + * uint32_t num_tx_ctx (number of TX queue context blocks)
> + * { raw struct igb_tx data } tx_ctx[num_tx_ctx]
> */
>
> +/* Maximum number of registers in the VF state slice */
> +#define IGB_VF_MAX_REGS 128
> +
> +/* Register offsets that constitute a VF's state slice */
> +static void igb_vf_reg_list(uint16_t vfn, uint32_t *offsets, int *count)
> +{
> + int n = 0;
> + int q0 = vfn;
> + int q1 = vfn + IGB_NUM_VM_POOLS;
> +
> + /* Per-VF control and interrupt registers */
> + offsets[n++] = E1000_PVTCTRL(vfn) >> 2;
> + offsets[n++] = E1000_PVTEICS(vfn) >> 2;
> + offsets[n++] = E1000_PVTEIMS(vfn) >> 2;
> + offsets[n++] = E1000_PVTEIMC(vfn) >> 2;
> + offsets[n++] = E1000_PVTEIAC(vfn) >> 2;
> + offsets[n++] = E1000_PVTEIAM(vfn) >> 2;
> + offsets[n++] = E1000_PVTEICR(vfn) >> 2;
> +
> + /* Per-VF statistics */
> + offsets[n++] = E1000_PVFGPRC(vfn) >> 2;
> + offsets[n++] = E1000_PVFGPTC(vfn) >> 2;
> + offsets[n++] = E1000_PVFGORC(vfn) >> 2;
> + offsets[n++] = E1000_PVFGOTC(vfn) >> 2;
> + offsets[n++] = E1000_PVFMPRC(vfn) >> 2;
> + offsets[n++] = E1000_PVFGPRLBC(vfn) >> 2;
> + offsets[n++] = E1000_PVFGPTLBC(vfn) >> 2;
> + offsets[n++] = E1000_PVFGORLBC(vfn) >> 2;
> + offsets[n++] = E1000_PVFGOTLBC(vfn) >> 2;
> +
> + /* Mailbox */
> + offsets[n++] = E1000_V2PMAILBOX(vfn) >> 2;
> + offsets[n++] = E1000_P2VMAILBOX(vfn) >> 2;
> +
> + /* Per-VF config */
> + offsets[n++] = E1000_VMOLR(vfn) >> 2;
> + offsets[n++] = E1000_VMVIR(vfn) >> 2;
> + offsets[n++] = E1000_PSRTYPE(vfn) >> 2;
> +
> + /*
> + * VF receive addresses (RA/RA2) are saved dynamically in
> + * igb_core_vf_save_state by scanning for entries whose pool
> + * bits match this VF - the PF driver chooses the RA slot.
> + */
> +
> + /* Interrupt routing */
> + offsets[n++] = (E1000_VTIVAR + vfn * 4) >> 2;
> + offsets[n++] = (E1000_VTIVAR_MISC + vfn * 4) >> 2;
> +
> + /*
> + * EITR (Extended Interrupt Throttle Register) - 3 vectors per VF.
> + * Each VF has 3 MSI-X vectors, each with its own EITR controlling
> + * interrupt coalescing. Without saving these, interrupt
> + * throttling resets to zero after migration which can cause
> + * interrupt storms or latency changes. VF N uses PF EITR indices
> + * (22 - N*3) .. (24 - N*3).
> + */
> + {
> + int eitr_base = 22 - vfn * 3;
> + offsets[n++] = E1000_EITR(eitr_base) >> 2;
> + offsets[n++] = E1000_EITR(eitr_base + 1) >> 2;
> + offsets[n++] = E1000_EITR(eitr_base + 2) >> 2;
> + }
> +
> + /* RX and TX queue registers for queues q0 and q1 */
> +#define ADD_QUEUE_REGS(q) do { \
> + offsets[n++] = E1000_RDBAL(q) >> 2; \
> + offsets[n++] = E1000_RDBAH(q) >> 2; \
> + offsets[n++] = E1000_RDLEN(q) >> 2; \
> + offsets[n++] = E1000_SRRCTL(q) >> 2; \
> + offsets[n++] = E1000_RDH(q) >> 2; \
> + offsets[n++] = E1000_RDT(q) >> 2; \
> + offsets[n++] = E1000_RXDCTL(q) >> 2; \
> + offsets[n++] = E1000_RXCTL(q) >> 2; \
> + offsets[n++] = E1000_RQDPC(q) >> 2; \
> + offsets[n++] = E1000_TDBAL(q) >> 2; \
> + offsets[n++] = E1000_TDBAH(q) >> 2; \
> + offsets[n++] = E1000_TDLEN(q) >> 2; \
> + offsets[n++] = E1000_TDH(q) >> 2; \
> + offsets[n++] = E1000_TDT(q) >> 2; \
> + offsets[n++] = E1000_TXDCTL(q) >> 2; \
> + offsets[n++] = E1000_TXCTL(q) >> 2; \
> + offsets[n++] = E1000_TDWBAL(q) >> 2; \
> + offsets[n++] = E1000_TDWBAH(q) >> 2; \
> +} while (0)
> +
> + ADD_QUEUE_REGS(q0);
> + ADD_QUEUE_REGS(q1);
> +#undef ADD_QUEUE_REGS
> +
> + g_assert(n <= IGB_VF_MAX_REGS);
> + *count = n;
> +}
> +
> +/*
> + * Scan RA and RA2 arrays for receive address entries assigned to
> + * this VF. The PF driver picks the RA slot, so we cannot use a
> + * fixed index - instead check each entry's pool bits.
> + */
> +static uint32_t *igb_core_vf_save_ra(IGBCore *core, uint16_t vfn,
> + uint32_t *p, int *total_regs)
> +{
> + uint32_t vf_pool_bit = E1000_RAH_POOL_1 << vfn;
> + static const struct {
> + uint32_t base;
> + int count;
> + } ra_banks[] = {
> + { RA, 16 },
> + { RA2, 8 },
> + };
> + int i, j;
> +
> + for (i = 0; i < ARRAY_SIZE(ra_banks); i++) {
> + for (j = 0; j < ra_banks[i].count; j++) {
> + uint32_t ral_off = ra_banks[i].base + j * 2;
> + uint32_t rah_off = ra_banks[i].base + j * 2 + 1;
> + uint32_t rah_val = core->mac[rah_off];
> +
> + if ((rah_val & E1000_RAH_AV) && (rah_val & vf_pool_bit)) {
> + *p++ = cpu_to_le32(ral_off);
> + *p++ = cpu_to_le32(core->mac[ral_off]);
> + *p++ = cpu_to_le32(rah_off);
> + *p++ = cpu_to_le32(rah_val);
> + *total_regs += 2;
> + }
> + }
> + }
> + return p;
> +}
> +
> +static uint32_t *igb_core_vf_save_tx_ctx(IGBCore *core, int queue,
> + uint32_t *p)
> +{
> + memcpy(p, &core->tx[queue], sizeof(struct igb_tx));
The state blob copies the entire host struct igb_tx using memcpy(). That
structure contains padding and a live NetTxPkt *. This exposes an L0
address and makes the ABI dependent on compiler layout, pointer width
and host endianness.
Regards,
Akihiko Odaki
> + return (uint32_t *)((uint8_t *)p + sizeof(struct igb_tx));
> +}
> +
> +static size_t igb_core_vf_state_max_size(int num_fixed_regs)
> +{
> + int max_ra_entries = 16 + 8; /* RA bank (16) + RA2 bank (8) */
> + int max_ra_regs = max_ra_entries * 2; /* RAL + RAH per entry */
> +
> + return 4 * sizeof(uint32_t) /* header */
> + + num_fixed_regs * 2 * sizeof(uint32_t) /* fixed reg pairs */
> + + max_ra_regs * 2 * sizeof(uint32_t) /* RA reg pairs */
> + + sizeof(uint32_t) /* num_tx_ctx */
> + + 2 * sizeof(struct igb_tx); /* TX context */
> +}
> +
> static int igb_core_vf_save_state(IgbVfState *s,
> void *buf, size_t buf_size)
> {
> - int size = 0;
> + IGBCore *core = igbvf_get_core(s);
> + uint32_t offsets[IGB_VF_MAX_REGS];
> + int num_regs, total_regs;
> + uint32_t *p = buf;
> + uint32_t *num_regs_p;
> + int i, size;
> + int q0 = s->vfn;
> + int q1 = s->vfn + IGB_NUM_VM_POOLS;
> +
> + /*
> + * Save PVT shadow registers (PVTEIMS/PVTEIAC/PVTEIAM) instead of
> + * extracting from PF aggregates - the L1 PF driver may have
> + * transiently cleared EIMS via EIMC. The load path ORs them back.
> + */
> + igb_vf_reg_list(s->vfn, offsets, &num_regs);
> +
> + if (!buf) {
> + return igb_core_vf_state_max_size(num_regs);
> + }
> +
> + if (igb_core_vf_state_max_size(num_regs) > buf_size) {
> + return -IGB_MIG_ERR_BAD_SIZE;
> + }
> +
> + /* Header: magic, version, vfn, num_regs (updated below) */
> + *p++ = cpu_to_le32(IGB_MIG_CAP_MAGIC);
> + *p++ = cpu_to_le32(1); /* version */
> + *p++ = cpu_to_le32(s->vfn);
> + num_regs_p = p;
> + *p++ = cpu_to_le32(num_regs);
> +
> + for (i = 0; i < num_regs; i++) {
> + *p++ = cpu_to_le32(offsets[i]);
> + *p++ = cpu_to_le32(core->mac[offsets[i]]);
> + }
> +
> + total_regs = num_regs;
> +
> + p = igb_core_vf_save_ra(core, s->vfn, p, &total_regs);
> +
> + *num_regs_p = cpu_to_le32(total_regs);
> +
> + /* TX context descriptors for this VF's two queues */
> + *p++ = cpu_to_le32(2); /* num_tx_ctx */
> + p = igb_core_vf_save_tx_ctx(core, q0, p);
> + p = igb_core_vf_save_tx_ctx(core, q1, p);
> +
> + size = (uint8_t *)p - (uint8_t *)buf;
>
> trace_igbvf_mig_save_state(s->vfn, size);
> return size;
> @@ -98,9 +308,74 @@ static int igb_core_vf_max_data_size(IgbVfState *s)
> return size;
> }
>
> +static const void *igb_core_vf_load_tx_ctx(IGBCore *core, int queue,
> + const void *data)
> +{
> + struct NetTxPkt *saved_pkt = core->tx[queue].tx_pkt;
> +
> + memcpy(&core->tx[queue], data, sizeof(struct igb_tx));
> + core->tx[queue].tx_pkt = saved_pkt;
> + return (const uint8_t *)data + sizeof(struct igb_tx);
> +}
> +
> static int igb_core_vf_load_state(IgbVfState *s,
> const void *buf, size_t size)
This function ignores size, does not require exactly two TX contexts and
accepts every register index below E1000E_MAC_SIZE. A crafted L1 blob
can therefore consume truncated data or overwrite PF-global and other-VF
register state.
It consumes ms->mig_data in igbvf_mig_data_xfer(), but it may be misaligned.
The blob also omits the 16-dword VF mailbox payload, embeds the source
VFN, and restores absolute/shared RA entries. It is neither complete nor
safely relocatable between VFs.
Regards,
Akihiko Odaki
> {
> + IGBCore *core = igbvf_get_core(s);
> + const uint32_t *p = buf;
> + uint32_t magic, version, saved_vfn, num_regs, num_tx;
> + int i;
> + int q0 = s->vfn;
> + int q1 = s->vfn + IGB_NUM_VM_POOLS;
> +
> + magic = le32_to_cpu(*p++);
> + version = le32_to_cpu(*p++);
> + saved_vfn = le32_to_cpu(*p++);
> + num_regs = le32_to_cpu(*p++);
> +
> + if (magic != IGB_MIG_CAP_MAGIC) {
> + return -IGB_MIG_ERR_BAD_MAGIC;
> + }
> + if (version != IGB_MIG_CAP_VERSION) {
> + return -IGB_MIG_ERR_BAD_VERSION;
> + }
> + if (saved_vfn != s->vfn) {
> + return -IGB_MIG_ERR_BAD_VFN;
> + }
> + if (num_regs > IGB_VF_MAX_REGS) {
> + return -IGB_MIG_ERR_BAD_SIZE;
> + }
> +
> + for (i = 0; i < num_regs; i++) {
> + uint32_t offset = le32_to_cpu(*p++);
> + uint32_t value = le32_to_cpu(*p++);
> +
> + if (offset < E1000E_MAC_SIZE) {
> + core->mac[offset] = value;
> +
> + /*
> + * Sync EITR to eitr_guest_value[] shadow array, stripping
> + * E1000_EITR_CNT_IGNR so guest register readback returns
> + * the correct value.
> + */
> + if (offset >= EITR0 && offset < EITR0 + IGB_INTR_NUM) {
> + core->eitr_guest_value[offset - EITR0] =
> + value & ~E1000_EITR_CNT_IGNR;
> + }
> + }
> + }
> +
> + num_tx = le32_to_cpu(*p++);
> + if (num_tx == 2) {
> + p = igb_core_vf_load_tx_ctx(core, q0, p);
> + p = igb_core_vf_load_tx_ctx(core, q1, p);
> + }
> +
> + /*
> + * MSI-X table/PBA is not saved - L1's VFIO reprograms it with
> + * destination-specific IRTE references after migration.
> + */
> +
> trace_igbvf_mig_load_state(s->vfn, (uint32_t)size);
> return 0;
> }
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH 05/11] igb: Add VF post-load fixups for live migration
2026-07-27 5:39 ` [RFC PATCH 05/11] igb: Add VF post-load fixups " Cédric Le Goater
@ 2026-07-27 7:53 ` Akihiko Odaki
0 siblings, 0 replies; 26+ messages in thread
From: Akihiko Odaki @ 2026-07-27 7:53 UTC (permalink / raw)
To: Cédric Le Goater, qemu-devel
Cc: Sriram Yagnaraman, Jason Wang, Alex Williamson,
Michael S . Tsirkin, Peter Xu, Avihai Horon
On 2026/07/27 14:39, Cédric Le Goater wrote:
> Add post-load fixups in igbvf_mig_load() to propagate VF interrupt
> state that the register load path bypasses by writing directly to
> mac[] without triggering register handler side effects.
>
> igb_core_vf_propagate_irqs() ORs PVT shadow values back into the PF
> aggregates (EIMS/EIAC/EIAM) and clears stale VF bits from EICR.
OR cannot clear stale destination bits. Furthermore, source EIMC can
clear the aggregate without updating PVTEIMS, so a disabled interrupt
may be restored as enabled.
>
> igb_core_vf_propagate_ivar() re-applies VTIVAR routing to the shared
> IVAR0 entries that the L1 PF driver may have overwritten after L0
> vmstate restore.
>
> Assisted-by: Claude
> Signed-off-by: Cédric Le Goater <clg@redhat.com>
> ---
> hw/net/igb_core.h | 2 ++
> hw/net/igb_core.c | 56 ++++++++++++++++++++++++++++++++++++++++++
> hw/net/igb_migration.c | 7 ++++++
> 3 files changed, 65 insertions(+)
>
> diff --git a/hw/net/igb_core.h b/hw/net/igb_core.h
> index 58d4f57c99bb..150567eb346d 100644
> --- a/hw/net/igb_core.h
> +++ b/hw/net/igb_core.h
> @@ -144,4 +144,6 @@ void
> igb_start_recv(IGBCore *core);
>
> IGBCore *igb_pf_get_core(void *pf);
> +void igb_core_vf_propagate_irqs(IGBCore *core, uint16_t vfn);
> +void igb_core_vf_propagate_ivar(IGBCore *core, uint16_t vfn);
> #endif
> diff --git a/hw/net/igb_core.c b/hw/net/igb_core.c
> index 45d8fd795b84..2565dd7f96d3 100644
> --- a/hw/net/igb_core.c
> +++ b/hw/net/igb_core.c
> @@ -4550,3 +4550,59 @@ igb_core_post_load(IGBCore *core)
>
> return 0;
> }
> +
> +/*
> + * Propagate VF interrupt state to PF aggregates after loading VF
> + * registers. The load path writes directly to mac[] bypassing the
> + * register handlers that OR VF bits into EIMS/EIAC/EIAM. Also clear
> + * stale VF bits in EICR that may have been set by packets arriving
> + * between PF vmstate restore and VF state load.
> + */
> +void igb_core_vf_propagate_irqs(IGBCore *core, uint16_t vfn)
> +{
> + uint32_t shift = 22 - vfn * IGBVF_MSIX_VEC_NUM;
> + uint32_t pvt_idx;
> +
> + pvt_idx = PVTEIMS0 + vfn * 0x40;
> + core->mac[EIMS] |= (core->mac[pvt_idx] & 0x7) << shift;
> + pvt_idx = PVTEIAC0 + vfn * 0x40;
> + core->mac[EIAC] |= (core->mac[pvt_idx] & 0x7) << shift;
> + pvt_idx = PVTEIAM0 + vfn * 0x40;
> + core->mac[EIAM] |= (core->mac[pvt_idx] & 0x7) << shift;
> +
> + core->mac[EICR] &= ~(0x7 << shift);
> +}
> +
> +/*
> + * Re-apply VTIVAR -> IVAR0 interrupt routing. The L1 PF driver
> + * may have overwritten the shared IVAR0 entries with its own
> + * queue routing after L0 vmstate restore.
> + */
> +void igb_core_vf_propagate_ivar(IGBCore *core, uint16_t vfn)
> +{
> + uint32_t vtivar = core->mac[VTIVAR + vfn];
> + int n;
> + uint8_t ent;
> + uint32_t mask;
> +
> + if (vtivar & E1000_IVAR_VALID) {
> + n = igb_ivar_entry_rx(vfn);
> + ent = E1000_IVAR_VALID |
> + (24 - vfn * IGBVF_MSIX_VEC_NUM - (2 - (vtivar & 0x7)));
> + mask = 0xffU << (8 * (n % 4));
> + core->mac[IVAR0 + n / 4] =
> + (core->mac[IVAR0 + n / 4] & ~mask) |
> + ((uint32_t)ent << (8 * (n % 4)));
> + }
> +
> + ent = vtivar >> 8;
> + if (ent & E1000_IVAR_VALID) {
> + n = igb_ivar_entry_tx(vfn);
> + ent = E1000_IVAR_VALID |
> + (24 - vfn * IGBVF_MSIX_VEC_NUM - (2 - (ent & 0x7)));
> + mask = 0xffU << (8 * (n % 4));
> + core->mac[IVAR0 + n / 4] =
> + (core->mac[IVAR0 + n / 4] & ~mask) |
> + ((uint32_t)ent << (8 * (n % 4)));
> + }
> +}
This function updates only valid entries. An invalid source route
consequently leaves a stale valid destination route. Replace or clear
each VF slice unconditionally.
Regards,
Akihiko Odaki
> diff --git a/hw/net/igb_migration.c b/hw/net/igb_migration.c
> index 61cf155a188d..4f123df6795d 100644
> --- a/hw/net/igb_migration.c
> +++ b/hw/net/igb_migration.c
> @@ -382,6 +382,7 @@ static int igb_core_vf_load_state(IgbVfState *s,
>
> static int igbvf_mig_load(IgbVfState *s, const void *buf, size_t size)
> {
> + IGBCore *core = igbvf_get_core(s);
> int ret;
>
> ret = igb_core_vf_load_state(s, buf, size);
> @@ -389,6 +390,12 @@ static int igbvf_mig_load(IgbVfState *s, const void *buf, size_t size)
> return ret;
> }
>
> + /*
> + * Post-load: sync VF interrupt and routing state to PF aggregates
> + */
> + igb_core_vf_propagate_irqs(core, s->vfn);
> + igb_core_vf_propagate_ivar(core, s->vfn);
> +
> return 0;
> }
>
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH 06/11] igb: Add dirty page tracking for IGBVF migration
2026-07-27 5:39 ` [RFC PATCH 06/11] igb: Add dirty page tracking for IGBVF migration Cédric Le Goater
@ 2026-07-27 8:15 ` Akihiko Odaki
0 siblings, 0 replies; 26+ messages in thread
From: Akihiko Odaki @ 2026-07-27 8:15 UTC (permalink / raw)
To: Cédric Le Goater, qemu-devel
Cc: Sriram Yagnaraman, Jason Wang, Alex Williamson,
Michael S . Tsirkin, Peter Xu, Avihai Horon
On 2026/07/27 14:39, Cédric Le Goater wrote:
> Extend the migration BAR with dirty page tracking registers:
>
> 0x020 DIRTY_PGSIZE (RW) - dirty tracking page granularity
> 0x024 DIRTY_CTRL (WO) - 0=DISABLE, 1=ENABLE, 2=QUERY
> 0x028 DIRTY_RANGE_IOVA_LO (WO) - low 32 bits of tracked range start
> 0x02C DIRTY_RANGE_IOVA_HI (WO) - high 32 bits of tracked range start
> 0x030 DIRTY_RANGE_SIZE (WO) - tracked range size in bytes
> 0x034 DIRTY_BUF_ADDR_LO (WO) - low 32 bits of shared buffer address
> 0x038 DIRTY_BUF_ADDR_HI (WO) - high 32 bits of shared buffer address
> 0x03C DIRTY_STATUS (RO) - result of last DIRTY_CTRL
>
> The CAPS register advertises the maximum number of ranges and
> supported page sizes. The driver enables tracking on IOVA ranges
> via DIRTY_CTRL and queries dirty bitmaps through a shared buffer.
> As for state transfers, dirty buffer DMA is performed through the
> PF device (pcie_sriov_get_pf).
>
> Each tracked range maintains its own bitmap scoped to its boundaries.
> DMA paths (TX/RX data, descriptor writeback) are instrumented to
> record the touched pages. This enables the PRE_COPY state where the
> driver iterates on dirty pages while the VM continues to run.
>
> Assisted-by: Claude
> Signed-off-by: Cédric Le Goater <clg@redhat.com>
> ---
> docs/system/devices/igb-migration.rst | 73 +++++
> hw/net/igb_core.h | 2 +
> hw/net/igb_migration.h | 82 ++++++
> hw/net/igb_core.c | 48 ++--
> hw/net/igb_migration.c | 367 +++++++++++++++++++++++++-
> hw/net/igbvf.c | 4 +
> hw/net/trace-events | 7 +
> 7 files changed, 562 insertions(+), 21 deletions(-)
>
> diff --git a/docs/system/devices/igb-migration.rst b/docs/system/devices/igb-migration.rst
> index 04c44ef072af..4a3a93f01dfc 100644
> --- a/docs/system/devices/igb-migration.rst
> +++ b/docs/system/devices/igb-migration.rst
> @@ -37,6 +37,14 @@ the following register layout::
> 0x014 DATA_XFER WO Trigger DMA save or DMA load
> 0x018 DATA_BUF_ADDR_LO WO Low 32 bits of state DMA buffer address
> 0x01C DATA_BUF_ADDR_HI WO High 32 bits of state DMA buffer address
> + 0x020 DIRTY_PGSIZE RW Dirty tracking page granularity
> + 0x024 DIRTY_CTRL WO 0=DISABLE, 1=ENABLE, 2=QUERY
> + 0x028 DIRTY_RANGE_IOVA_LO WO Low 32 bits of tracked range start
> + 0x02C DIRTY_RANGE_IOVA_HI WO High 32 bits of tracked range start
> + 0x030 DIRTY_RANGE_SIZE WO Tracked range size in bytes
> + 0x034 DIRTY_BUF_ADDR_LO WO Low 32 bits of shared buffer address
> + 0x038 DIRTY_BUF_ADDR_HI WO High 32 bits of shared buffer address
> + 0x03C DIRTY_STATUS RO Result of last DIRTY_CTRL (0=OK, 1-5=error)
>
> State transitions follow the VFIO migration state machine: the driver
> writes to ``DEVICE_STATE`` to move between states and reads ``STATUS``
> @@ -61,3 +69,68 @@ code identifying the failure::
> 4 BAD_VFN VF number mismatch (source != destination)
> 5 DMA_FAILED DMA transfer to/from state buffer failed
> 6 NO_BUFFER DATA_XFER without buffer address set
> +
> +Dirty page tracking
> +~~~~~~~~~~~~~~~~~~~
> +
> +The migration interface supports per-VF dirty page tracking, advertised
> +by the ``F_DIRTY`` flag in ``CAPS``. This allows the variant driver to
> +enter ``PRE_COPY`` state (``DEVICE_STATE`` = 5) while the VM continues
> +to run, iterating on dirty pages to reduce the final stop-and-copy
> +window.
> +
> +The device maintains one dirty tracking engine per range, each with its
> +own bitmap scoped to the range boundaries. The ``CAPS`` register
> +advertises the maximum number of ranges (``max_ranges`` in bits [11:8])
> +and supported page sizes (bits [31:12]).
> +
> +Dirty tracking is controlled through the ``DIRTY_CTRL`` register:
> +
> +- **ENABLE** (1): the driver programs a tracked range via
> + ``DIRTY_RANGE_IOVA_LO/HI`` + ``DIRTY_RANGE_SIZE`` then writes
> + ``DIRTY_CTRL=ENABLE``. The device allocates a fixed-size bitmap for
> + the range and begins recording pages touched by DMA (TX data, RX
> + data, descriptor writeback). The page granularity is set by
> + ``DIRTY_PGSIZE`` (default 4096). After each ENABLE the driver reads
> + ``DIRTY_STATUS`` to check for errors.
> +- **DISABLE** (0): tears down all ranges and stops tracking.
> +- **QUERY** (2): the driver writes (IOVA, size, page_size) into a
> + shared buffer registered via ``DIRTY_BUF_ADDR_LO/HI`` (PF DMA
> + address, as for state transfers), then writes
> + ``DIRTY_CTRL=QUERY``. The device finds the matching range, copies
> + the dirty bitmap into the shared buffer, clears the tracked bits,
> + and sets the buffer's completion status.
> +
> +``DIRTY_STATUS`` values after each ``DIRTY_CTRL`` write::
> +
> + 0 OK Success
> + 1 TOO_MANY_RANGES Exceeds max_ranges from CAPS
> + 2 BAD_RANGE Invalid range (zero size)
> + 3 BAD_PGSIZE Invalid or misaligned page size
> + 4 NOT_ENABLED Query without prior enable
> + 5 NO_BUFFER Query without shared buffer
> +
> +Dirty query shared buffer
> +~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +The shared buffer used for dirty queries is registered via
> +``DIRTY_BUF_ADDR_LO/HI`` (PF DMA address). It is cache-line aligned
> +(64 bytes) to separate driver-written request fields from
> +device-written completion fields::
> +
> + Offset Field Written by Description
> + 0x00 iova driver Query range start
> + 0x08 size driver Query range size
> + 0x10 page_size driver Page granularity
> + 0x14 flags driver Reserved (must be 0)
> + 0x18 reserved[10] - Pad to 64-byte cache line
> + 0x40 status device 0 = pending, 1 = complete
> + 0x44 bitmap_size device Bytes written to bitmap
> + 0x48 dirty_page_count device Number of set bits in bitmap
> + 0x4C reserved[12] - Pad to 64-byte cache line
> + 0x80 bitmap[] device Dirty page bitmap
> +
> +The driver fills the request fields, issues ``DIRTY_CTRL=QUERY``, and
> +polls ``status`` for completion. The device reads the request, writes
> +the dirty bitmap and completion fields via DMA, then sets
> +``status = 1``.
> diff --git a/hw/net/igb_core.h b/hw/net/igb_core.h
> index 150567eb346d..50cee2c7683c 100644
> --- a/hw/net/igb_core.h
> +++ b/hw/net/igb_core.h
> @@ -83,6 +83,8 @@ struct IGBCore {
> struct NetTxPkt *tx_pkt;
> } tx[IGB_NUM_QUEUES];
>
> + IGBVfDirtyState vf_dirty[IGB_MAX_VF_FUNCTIONS];
> +
> struct NetRxPkt *rx_pkt;
>
> bool has_vnet;
> diff --git a/hw/net/igb_migration.h b/hw/net/igb_migration.h
> index 739a189810b0..a2e7b363cb04 100644
> --- a/hw/net/igb_migration.h
> +++ b/hw/net/igb_migration.h
> @@ -27,6 +27,12 @@
> #define IGB_MIG_CAP_F_STATE (1u << 0) /* device state serialization */
> #define IGB_MIG_CAP_F_DIRTY (1u << 1) /* dirty page tracking */
>
> +/* MIG_CAPS register layout (read-only, offset 0x008) */
> +#define IGB_MIG_CAPS_MAX_RANGES_SHIFT 8
> +#define IGB_MIG_CAPS_MAX_RANGES_MASK (0xfu << 8) /* bits [11:8] */
> +#define IGB_MIG_CAPS_MAX_RANGES 4
> +#define IGB_MIG_CAPS_PGSIZES_MASK 0xfffff000u /* bits [31:12] */
> +
> #define IGB_MIG_CAP_SIZE 16
> #define IGB_MIG_CAP_OFF_MAGIC 4 /* offset within cap for magic field */
> #define IGB_MIG_CAP_OFF_BARID 8 /* offset within cap for BAR id */
> @@ -49,6 +55,14 @@
> #define IGB_MIG_DATA_XFER 0x014
> #define IGB_MIG_DATA_BUF_ADDR_LO 0x018
> #define IGB_MIG_DATA_BUF_ADDR_HI 0x01C
> +#define IGB_MIG_DIRTY_PGSIZE 0x020
> +#define IGB_MIG_DIRTY_CTRL 0x024
> +#define IGB_MIG_DIRTY_RANGE_IOVA_LO 0x028
> +#define IGB_MIG_DIRTY_RANGE_IOVA_HI 0x02C
> +#define IGB_MIG_DIRTY_RANGE_SIZE 0x030
> +#define IGB_MIG_DIRTY_BUF_ADDR_LO 0x034
> +#define IGB_MIG_DIRTY_BUF_ADDR_HI 0x038
> +#define IGB_MIG_DIRTY_STATUS 0x03C
>
> /* DEVICE_STATE values - mirrors VFIO migration states */
> #define IGB_MIG_STATE_ERROR 0
> @@ -75,6 +89,50 @@
> #define IGB_MIG_ERR_DMA_FAILED 5
> #define IGB_MIG_ERR_NO_BUFFER 6
>
> +/*
> + * DIRTY_CTRL register values. Write one of these to control
> + * the dirty page tracking state machine.
> + */
> +#define IGB_MIG_DIRTY_CTRL_DISABLE 0
> +#define IGB_MIG_DIRTY_CTRL_ENABLE 1
> +#define IGB_MIG_DIRTY_CTRL_QUERY 2 /* query-and-clear */
> +
> +/* DIRTY_STATUS register values (read-only, cleared on next DIRTY_CTRL write) */
> +#define IGB_MIG_DIRTY_STATUS_OK 0
> +#define IGB_MIG_DIRTY_STATUS_TOO_MANY_RANGES 1
> +#define IGB_MIG_DIRTY_STATUS_BAD_RANGE 2
> +#define IGB_MIG_DIRTY_STATUS_BAD_PGSIZE 3
> +#define IGB_MIG_DIRTY_STATUS_NOT_ENABLED 4
> +#define IGB_MIG_DIRTY_STATUS_NO_BUFFER 5
> +
> +#define IGB_MIG_DIRTY_DEFAULT_PGSIZE 4096
> +
> +/*
> + * Dirty query shared buffer layout - DMA between driver and device.
> + * The driver writes request fields, kicks DIRTY_CTRL=QUERY, and the
> + * device reads the request, fills bitmap + completion via DMA. Each
> + * section is cache-line aligned (64 bytes).
> + */
> +struct igb_mig_dirty_query {
> + /* Cache line 0: request (written by driver) */
> + uint64_t iova;
> + uint64_t size;
> + uint32_t page_size;
> + uint32_t flags;
> + uint32_t reserved0[10];
> +
> + /* Cache line 1: completion (written by device) */
> + uint32_t status;
> + uint32_t bitmap_size;
> + uint32_t dirty_page_count;
> + uint32_t reserved1[12];
> +
> + /* Cache line 2+: bitmap (written by device) */
> + uint8_t bitmap[];
The bitmap is at offset 0x7c, not the promised 0x80.
> +};
> +
> +#define IGB_MIG_DIRTY_STATUS_COMPLETE 1
> +
> typedef struct IgbVfMigState {
> bool migration_cap;
> MemoryRegion mig_bar;
> @@ -84,6 +142,12 @@ typedef struct IgbVfMigState {
> uint8_t mig_data[IGB_VF_STATE_MAX_SIZE];
> uint32_t mig_data_size;
> uint64_t mig_data_buf_addr;
> +
> + uint32_t mig_dirty_pgsize;
> + uint64_t mig_dirty_range_iova;
> + uint32_t mig_dirty_range_size;
> + uint64_t mig_dirty_buf_addr;
> + uint32_t mig_dirty_status;
> } IgbVfMigState;
>
> typedef struct IgbVfState IgbVfState;
> @@ -92,4 +156,22 @@ bool igbvf_add_migration_cap(PCIDevice *dev, Error **errp);
> void igbvf_mig_bar_init(IgbVfState *s);
> void igbvf_mig_state_reset(IgbVfState *s);
>
> +typedef struct IGBVfDirtyRange {
> + uint64_t iova;
> + uint64_t size;
> + uint64_t page_size;
> + unsigned long *bitmap;
> + uint64_t nbits;
> +} IGBVfDirtyRange;
> +
> +typedef struct IGBVfDirtyState {
> + uint32_t num_ranges;
> + IGBVfDirtyRange ranges[IGB_MIG_CAPS_MAX_RANGES];
> +} IGBVfDirtyState;
> +
> +typedef struct IGBCore IGBCore;
> +void igb_core_dirty_track_dma(IGBCore *core, int vfn,
> + dma_addr_t addr, dma_addr_t len);
> +void igb_core_vf_dirty_disable(IgbVfState *s);
> +
> #endif
> diff --git a/hw/net/igb_core.c b/hw/net/igb_core.c
> index 2565dd7f96d3..03c349575c0a 100644
> --- a/hw/net/igb_core.c
> +++ b/hw/net/igb_core.c
> @@ -824,6 +824,14 @@ igb_rx_ring_init(IGBCore *core, E1000E_RxRing *rxr, int idx)
> rxr->i = &i[idx];
> }
>
> +static inline void
> +igb_pci_dma_write(IGBCore *core, PCIDevice *dev, int vfn,
> + dma_addr_t addr, const void *buf, dma_addr_t len)
> +{
> + pci_dma_write(dev, addr, buf, len);
> + igb_core_dirty_track_dma(core, vfn, addr, len);
> +}
> +
> static uint32_t
> igb_txdesc_writeback(IGBCore *core, dma_addr_t base,
> union e1000_adv_tx_desc *tx_desc,
> @@ -847,13 +855,15 @@ igb_txdesc_writeback(IGBCore *core, dma_addr_t base,
>
> if (tdwba & 1) {
> uint32_t buffer = cpu_to_le32(core->mac[txi->dh]);
> - pci_dma_write(d, tdwba & ~3, &buffer, sizeof(buffer));
> + igb_pci_dma_write(core, d, txi->idx % IGB_NUM_VM_POOLS,
> + tdwba & ~3, &buffer, sizeof(buffer));
> } else {
> uint32_t status = le32_to_cpu(tx_desc->wb.status) | E1000_TXD_STAT_DD;
>
> tx_desc->wb.status = cpu_to_le32(status);
> - pci_dma_write(d, base + offsetof(union e1000_adv_tx_desc, wb),
> - &tx_desc->wb, sizeof(tx_desc->wb));
> + igb_pci_dma_write(core, d, txi->idx % IGB_NUM_VM_POOLS,
> + base + offsetof(union e1000_adv_tx_desc, wb),
> + &tx_desc->wb, sizeof(tx_desc->wb));
> }
>
> return igb_tx_wb_eic(core, txi->idx);
> @@ -1264,6 +1274,7 @@ typedef struct IGBPacketRxDMAState {
> size_t iov_ofs;
> bool do_ps;
> bool is_first;
> + int vfn;
> IGBBAState bastate;
> hwaddr ba[IGB_MAX_PS_BUFFERS];
> IGBSplitDescriptorData ps_desc_data;
> @@ -1590,7 +1601,8 @@ igb_write_rx_descr(IGBCore *core,
>
> static inline void
> igb_pci_dma_write_rx_desc(IGBCore *core, PCIDevice *dev, dma_addr_t addr,
> - union e1000_rx_desc_union *desc, dma_addr_t len)
> + union e1000_rx_desc_union *desc, dma_addr_t len,
> + int vfn)
> {
> if (igb_rx_use_legacy_descriptor(core)) {
> struct e1000_rx_desc *d = &desc->legacy;
> @@ -1598,11 +1610,12 @@ igb_pci_dma_write_rx_desc(IGBCore *core, PCIDevice *dev, dma_addr_t addr,
> uint8_t status = d->status;
>
> d->status &= ~E1000_RXD_STAT_DD;
> - pci_dma_write(dev, addr, desc, len);
> + igb_pci_dma_write(core, dev, vfn, addr, desc, len);
>
> if (status & E1000_RXD_STAT_DD) {
> d->status = status;
> - pci_dma_write(dev, addr + offset, &status, sizeof(status));
> + igb_pci_dma_write(core, dev, vfn,
> + addr + offset, &status, sizeof(status));
> }
> } else {
> union e1000_adv_rx_desc *d = &desc->adv;
> @@ -1611,11 +1624,12 @@ igb_pci_dma_write_rx_desc(IGBCore *core, PCIDevice *dev, dma_addr_t addr,
> uint32_t status = d->wb.upper.status_error;
>
> d->wb.upper.status_error &= ~E1000_RXD_STAT_DD;
> - pci_dma_write(dev, addr, desc, len);
> + igb_pci_dma_write(core, dev, vfn, addr, desc, len);
>
> if (status & E1000_RXD_STAT_DD) {
> d->wb.upper.status_error = status;
> - pci_dma_write(dev, addr + offset, &status, sizeof(status));
> + igb_pci_dma_write(core, dev, vfn,
> + addr + offset, &status, sizeof(status));
> }
> }
> }
> @@ -1737,9 +1751,9 @@ igb_write_hdr_frag_to_rx_buffers(IGBCore *core,
> {
> assert(data_len <= pdma_st->rx_desc_header_buf_size -
> pdma_st->bastate.written[0]);
> - pci_dma_write(d,
> - pdma_st->ba[0] + pdma_st->bastate.written[0],
> - data, data_len);
> + igb_pci_dma_write(core, d, pdma_st->vfn,
> + pdma_st->ba[0] + pdma_st->bastate.written[0],
> + data, data_len);
> pdma_st->bastate.written[0] += data_len;
> pdma_st->bastate.cur_idx = 1;
> }
> @@ -1804,10 +1818,10 @@ igb_write_payload_frag_to_rx_buffers(IGBCore *core,
> data,
> bytes_to_write);
>
> - pci_dma_write(d,
> - pdma_st->ba[pdma_st->bastate.cur_idx] +
> - pdma_st->bastate.written[pdma_st->bastate.cur_idx],
> - data, bytes_to_write);
> + igb_pci_dma_write(core, d, pdma_st->vfn,
> + pdma_st->ba[pdma_st->bastate.cur_idx] +
> + pdma_st->bastate.written[pdma_st->bastate.cur_idx],
> + data, bytes_to_write);
>
> pdma_st->bastate.written[pdma_st->bastate.cur_idx] += bytes_to_write;
> data += bytes_to_write;
> @@ -1908,6 +1922,7 @@ igb_write_packet_to_guest(IGBCore *core, struct NetRxPkt *pkt,
>
> rxi = rxr->i;
> rx_desc_len = core->rx_desc_len;
> + pdma_st.vfn = rxi->idx % IGB_NUM_VM_POOLS;
> pdma_st.rx_desc_packet_buf_size = igb_rxbufsize(core, rxi);
> pdma_st.rx_desc_header_buf_size = igb_rxhdrbufsize(core, rxi);
> pdma_st.iov = net_rx_pkt_get_iovec(pkt);
> @@ -1944,7 +1959,8 @@ igb_write_packet_to_guest(IGBCore *core, struct NetRxPkt *pkt,
> etqf, ts,
> &pdma_st,
> rxi);
> - igb_pci_dma_write_rx_desc(core, d, base, &desc, rx_desc_len);
> + igb_pci_dma_write_rx_desc(core, d, base, &desc, rx_desc_len,
> + rxi->idx % IGB_NUM_VM_POOLS);
> igb_ring_advance(core, rxi, rx_desc_len / E1000_MIN_RX_DESC_LEN);
> } while (pdma_st.desc_offset < pdma_st.total_size);
>
> diff --git a/hw/net/igb_migration.c b/hw/net/igb_migration.c
> index 4f123df6795d..9a9ce902bb4a 100644
> --- a/hw/net/igb_migration.c
> +++ b/hw/net/igb_migration.c
> @@ -8,6 +8,8 @@
>
> #include "qemu/osdep.h"
> #include "qemu/log.h"
> +#include "qemu/units.h"
> +#include "qemu/bitmap.h"
> #include "hw/pci/pci_device.h"
> #include "hw/pci/pcie.h"
> #include "net/eth.h"
> @@ -69,7 +71,7 @@ bool igbvf_add_migration_cap(PCIDevice *dev, Error **errp)
> pci_set_long(dev->config + offset + IGB_MIG_CAP_OFF_BARID,
> IGB_MIG_BAR_IDX);
> pci_set_long(dev->config + offset + IGB_MIG_CAP_OFF_FLAGS,
> - IGB_MIG_CAP_F_STATE);
> + IGB_MIG_CAP_F_STATE | IGB_MIG_CAP_F_DIRTY);
>
> trace_igbvf_mig_cap_add(pcie_sriov_vf_number(dev), offset);
> return true;
> @@ -399,6 +401,161 @@ static int igbvf_mig_load(IgbVfState *s, const void *buf, size_t size)
> return 0;
> }
>
> +/*
> + * =====================================================================
> + * Per-VF dirty page tracking
> + * =====================================================================
> + *
> + * All VF DMA writes in igb_core.c go through igb_pci_dma_write(),
> + * which calls igb_core_dirty_track_dma() to mark the target page in a
> + * per-range bitmap before performing the actual DMA.
> + *
> + * The IGBCore::vf_dirty[] bitmaps live in IGBCore so they are easily
> + * accessible from the core TX and RX paths without reaching back into
> + * VF state.
> + */
> +
> +void igb_core_dirty_track_dma(IGBCore *core, int vfn,
> + dma_addr_t addr, dma_addr_t len)
> +{
> + IGBVfDirtyState *ds = &core->vf_dirty[vfn];
> + bool matched = false;
> + uint32_t i;
> +
> + if (!ds->num_ranges) {
> + return;
> + }
> +
> + trace_igb_core_dirty_track_dma(vfn, addr, len);
> +
> + for (i = 0; i < ds->num_ranges; i++) {
> + IGBVfDirtyRange *r = &ds->ranges[i];
> + uint64_t r_end = r->iova + r->size;
> + uint64_t dma_end = addr + len;
> + uint64_t start, end, start_page, end_page, page;
> +
> + if (addr >= r_end || dma_end <= r->iova) {
> + continue;
> + }
> +
> + matched = true;
> + start = MAX(addr, r->iova);
> + end = MIN(dma_end, r_end);
> +
> + start_page = (start - r->iova) / r->page_size;
> + end_page = (end - 1 - r->iova) / r->page_size;
> +
> + for (page = start_page; page <= end_page; page++) {
> + if (page < r->nbits) {
> + set_bit(page, r->bitmap);
> + }
> + }
> + }
> +
> + if (!matched) {
> + trace_igb_core_dirty_track_dma_drop(vfn, addr, len);
> + }
> +}
> +
> +static IGBVfDirtyState *igb_core_vf_dirty_state(IgbVfState *s)
> +{
> + IGBCore *core = igbvf_get_core(s);
> + return &core->vf_dirty[s->vfn];
> +}
> +
> +#define IGB_MIG_DIRTY_MAX_PAGES ((256 * GiB) / (4 * KiB))
> +
> +static uint32_t igb_core_vf_dirty_enable(IgbVfState *s, uint64_t pgsize,
> + uint64_t range_iova, uint64_t range_size)
> +{
> + IGBVfDirtyState *ds = igb_core_vf_dirty_state(s);
> + IGBVfDirtyRange *r;
> +
> + if (ds->num_ranges >= IGB_MIG_CAPS_MAX_RANGES) {
> + return IGB_MIG_DIRTY_STATUS_TOO_MANY_RANGES;
> + }
> +
> + if (!range_size) {
> + return IGB_MIG_DIRTY_STATUS_BAD_RANGE;
> + }
> +
> + if (!pgsize || (range_iova % pgsize) || (range_size % pgsize)) {
> + return IGB_MIG_DIRTY_STATUS_BAD_PGSIZE;
> + }
> +
> + if (range_size / pgsize > IGB_MIG_DIRTY_MAX_PAGES) {
> + return IGB_MIG_DIRTY_STATUS_BAD_RANGE;
> + }
> +
> + r = &ds->ranges[ds->num_ranges];
> + r->iova = range_iova;
> + r->size = range_size;
> + r->page_size = pgsize;
> + r->nbits = range_size / pgsize;
> + r->bitmap = bitmap_new(r->nbits);
> + ds->num_ranges++;
> + return IGB_MIG_DIRTY_STATUS_OK;
> +}
> +
> +void igb_core_vf_dirty_disable(IgbVfState *s)
> +{
> + IGBVfDirtyState *ds = igb_core_vf_dirty_state(s);
> + uint32_t i;
> +
> + for (i = 0; i < ds->num_ranges; i++) {
> + IGBVfDirtyRange *r = &ds->ranges[i];
> +
> + g_free(r->bitmap);
> + r->bitmap = NULL;
> + r->nbits = 0;
> + }
> + ds->num_ranges = 0;
> +}
> +
> +static bool igb_core_vf_dirty_enabled(IgbVfState *s)
> +{
> + IGBVfDirtyState *ds = igb_core_vf_dirty_state(s);
> + return !!ds->num_ranges;
> +}
> +
> +static bool igb_core_vf_dirty_query(IgbVfState *s,
> + void *buf, size_t buf_size, size_t *out_size,
> + uint64_t range_iova, uint64_t range_size)
> +{
> + IGBVfDirtyState *ds = igb_core_vf_dirty_state(s);
> + uint32_t i;
> +
> + for (i = 0; i < ds->num_ranges; i++) {
> + IGBVfDirtyRange *r = &ds->ranges[i];
> + uint64_t start_page, range_pages, count;
> +
> + if (range_iova < r->iova ||
> + range_iova + range_size > r->iova + r->size) {
> + continue;
> + }
> +
> + start_page = (range_iova - r->iova) / r->page_size;
> + range_pages = (uint64_t)range_size / r->page_size;
> + count = MIN(range_pages, (uint64_t)buf_size * 8);
> +
> + memset(buf, 0, buf_size);
> +
> + if (start_page < r->nbits) {
> + uint64_t avail = r->nbits - start_page;
> + uint64_t n = MIN(count, avail);
> +
> + bitmap_copy_with_src_offset(buf, r->bitmap, start_page, n);
> + bitmap_clear(r->bitmap, start_page, n);
> + }
> +
> + *out_size = bitmap_empty(buf, count) ? 0 : DIV_ROUND_UP(count, 8);
> + return true;
> + }
> +
> + *out_size = 0;
> + return false;
> +}
> +
> /* ================================================================
> * Migration BAR register read/write handlers
> * ================================================================ */
I forgot to point out this, but this does not match with QEMU's comment
style:
> Multiline comment blocks should have a row of stars on the left,
> and the initial /``*`` and terminating ``*``/ both on their own lines:
>
> .. code-block:: c
>
> /*
> * like
> * this
> */
Please also avoid decoration with "=". It has little benefit while
updating the comment a bit troublesome.
> @@ -410,25 +567,53 @@ static bool igbvf_mig_set_state(IgbVfState *s, uint32_t new_state)
> int ret;
>
> switch (new_state) {
> + case IGB_MIG_STATE_PRE_COPY:
> + if (old != IGB_MIG_STATE_RUNNING) {
> + return false;
> + }
> + /*
> + * Take an initial snapshot so DATA_SIZE reflects the actual
> + * state size and DATA_AVAIL is set for the driver.
> + */
> + ret = igb_core_vf_save_state(s, ms->mig_data, sizeof(ms->mig_data));
> + if (ret < 0) {
> + ms->mig_error = -ret;
> + ms->mig_state = IGB_MIG_STATE_ERROR;
> + return false;
> + }
> + ms->mig_data_size = ret;
> + break;
> +
> case IGB_MIG_STATE_STOP:
> if (old != IGB_MIG_STATE_RUNNING &&
> old != IGB_MIG_STATE_STOP_COPY &&
> + old != IGB_MIG_STATE_PRE_COPY &&
> old != IGB_MIG_STATE_RESUMING &&
> old != IGB_MIG_STATE_ERROR) {
> return false;
> }
> /* Restore DATA_SIZE to max, same as at reset */
> ms->mig_data_size = igb_core_vf_max_data_size(s);
> + if (old == IGB_MIG_STATE_PRE_COPY ||
> + old == IGB_MIG_STATE_STOP_COPY ||
> + old == IGB_MIG_STATE_ERROR) {
> + igb_core_vf_dirty_disable(s);
> + }
> break;
>
> case IGB_MIG_STATE_RUNNING:
> - if (old != IGB_MIG_STATE_STOP) {
> + if (old != IGB_MIG_STATE_STOP &&
> + old != IGB_MIG_STATE_PRE_COPY) {
> return false;
> }
> + if (old == IGB_MIG_STATE_PRE_COPY) {
> + igb_core_vf_dirty_disable(s);
> + }
> break;
>
> case IGB_MIG_STATE_STOP_COPY:
> - if (old != IGB_MIG_STATE_STOP) {
> + if (old != IGB_MIG_STATE_STOP &&
> + old != IGB_MIG_STATE_PRE_COPY) {
> return false;
> }
> ret = igb_core_vf_save_state(s, ms->mig_data, sizeof(ms->mig_data));
> @@ -466,7 +651,8 @@ static uint32_t igbvf_mig_get_status(IgbVfState *s)
> if (ms->mig_state == IGB_MIG_STATE_ERROR) {
> status |= IGB_MIG_STATUS_ERR(ms->mig_error);
> }
> - if (ms->mig_state == IGB_MIG_STATE_STOP_COPY && ms->mig_data_size > 0) {
> + if ((ms->mig_state == IGB_MIG_STATE_STOP_COPY ||
> + ms->mig_state == IGB_MIG_STATE_PRE_COPY) && ms->mig_data_size > 0) {
> status |= IGB_MIG_STATUS_DATA_AVAIL;
> }
>
> @@ -486,6 +672,30 @@ static void igbvf_mig_data_xfer(IgbVfState *s, uint32_t val)
> }
>
> switch (ms->mig_state) {
> + case IGB_MIG_STATE_PRE_COPY:
> + /*
> + * Re-snapshot device state so the driver can read a fresh
> + * copy on each pre-copy iteration.
> + */
> + ret = igb_core_vf_save_state(s, ms->mig_data, sizeof(ms->mig_data));
> + if (ret < 0) {
> + ms->mig_error = -ret;
> + ms->mig_state = IGB_MIG_STATE_ERROR;
> + return;
> + }
> + ms->mig_data_size = ret;
> + r = pci_dma_write(pcie_sriov_get_pf(PCI_DEVICE(s)),
> + ms->mig_data_buf_addr,
> + ms->mig_data, ms->mig_data_size);
> + if (r != MEMTX_OK) {
> + qemu_log_mask(LOG_GUEST_ERROR,
> + "igbvf: VF%u state write failed at 0x%" PRIx64 "\n",
> + s->vfn, ms->mig_data_buf_addr);
> + ms->mig_error = IGB_MIG_ERR_DMA_FAILED;
> + ms->mig_state = IGB_MIG_STATE_ERROR;
> + }
> + break;
> +
> case IGB_MIG_STATE_STOP_COPY:
> /* Save: DMA-write serialized state to driver buffer */
> r = pci_dma_write(pcie_sriov_get_pf(PCI_DEVICE(s)),
> @@ -547,7 +757,9 @@ static uint64_t igbvf_mig_read(void *opaque, hwaddr addr, unsigned size)
> val = igbvf_mig_get_status(s);
> break;
> case IGB_MIG_CAPS:
> - val = IGB_MIG_CAP_F_STATE;
> + val = IGB_MIG_CAP_F_STATE | IGB_MIG_CAP_F_DIRTY |
> + (IGB_MIG_CAPS_MAX_RANGES << IGB_MIG_CAPS_MAX_RANGES_SHIFT) |
> + (1u << 12); /* 4K page size supported */
> break;
> case IGB_MIG_VERSION:
> val = IGB_MIG_CAP_VERSION;
> @@ -555,6 +767,19 @@ static uint64_t igbvf_mig_read(void *opaque, hwaddr addr, unsigned size)
> case IGB_MIG_DATA_SIZE:
> val = ms->mig_data_size;
> break;
> + case IGB_MIG_DIRTY_PGSIZE:
> + val = ms->mig_dirty_pgsize ? ms->mig_dirty_pgsize
> + : IGB_MIG_DIRTY_DEFAULT_PGSIZE;
> + break;
> + case IGB_MIG_DIRTY_BUF_ADDR_LO:
> + val = (uint32_t)ms->mig_dirty_buf_addr;
> + break;
> + case IGB_MIG_DIRTY_BUF_ADDR_HI:
> + val = (uint32_t)(ms->mig_dirty_buf_addr >> 32);
> + break;
> + case IGB_MIG_DIRTY_STATUS:
> + val = ms->mig_dirty_status;
> + break;
> default:
> qemu_log_mask(LOG_GUEST_ERROR,
> "igbvf: VF%u bad migration BAR read at 0x%"
> @@ -567,6 +792,106 @@ static uint64_t igbvf_mig_read(void *opaque, hwaddr addr, unsigned size)
> return val;
> }
>
> +static uint32_t igbvf_mig_dirty_count(const void *bitmap, size_t size)
> +{
> + const unsigned long *p = bitmap;
> + size_t n = size / sizeof(unsigned long);
> + uint32_t count = 0;
> +
> + for (size_t i = 0; i < n; i++) {
> + count += ctpopl(p[i]);
> + }
> + return count;
> +}
> +
> +static void igbvf_mig_dirty_query(IgbVfState *s, uint64_t pgsize)
> +{
> + IgbVfMigState *ms = &s->mig;
> + PCIDevice *dev = pcie_sriov_get_pf(PCI_DEVICE(s));
> + uint64_t buf_addr = ms->mig_dirty_buf_addr;
> + uint64_t range_iova = 0, range_size = 0;
> + uint32_t bmp_bytes;
> + size_t out_size;
> + g_autofree void *bitmap = NULL;
> + bool valid;
> +
> + ldq_le_pci_dma(dev,
> + buf_addr + offsetof(struct igb_mig_dirty_query, iova),
> + &range_iova, MEMTXATTRS_UNSPECIFIED);
> + ldq_le_pci_dma(dev,
> + buf_addr + offsetof(struct igb_mig_dirty_query, size),
> + &range_size, MEMTXATTRS_UNSPECIFIED);
> +
> + bmp_bytes = DIV_ROUND_UP(range_size / pgsize, 8);
> + bitmap = g_malloc0(bmp_bytes);
The bitmap is later used with QEMU's `unsigned long` bitmap API. A
one-page query allocates one byte but accesses an eight-byte word on a
64-bit host: guest-triggerable heap corruption.
The untrusted 64-bit query size is allocated before it is matched
against an enabled range. For example, a 64-TiB request with 4-KiB pages
asks g_malloc0() for 2 GiB, allowing L1 to abort QEMU through memory
exhaustion.
> +
> + valid = igb_core_vf_dirty_query(s, bitmap, bmp_bytes, &out_size,
> + range_iova, range_size);
> +
> + if (valid && out_size) {
> + if (pci_dma_write(dev,
> + buf_addr + offsetof(struct igb_mig_dirty_query, bitmap),
> + bitmap, out_size)) {
> + qemu_log_mask(LOG_GUEST_ERROR,
> + "igbvf: VF%u dirty bitmap write failed at 0x%" PRIx64 "\n",
> + s->vfn, buf_addr);
> + valid = false;
> + }
> + }
Dirty bits are cleared before the bitmap is DMA-written. A DMA failure
permanently loses dirty information and can corrupt the migrated guest.
> +
> + uint32_t dirty_pages = valid ? igbvf_mig_dirty_count(bitmap, out_size) : 0;
> +
> + stl_le_pci_dma(dev,
> + buf_addr + offsetof(struct igb_mig_dirty_query, bitmap_size),
> + valid ? out_size : 0, MEMTXATTRS_UNSPECIFIED);
> + stl_le_pci_dma(dev,
> + buf_addr + offsetof(struct igb_mig_dirty_query, dirty_page_count),
> + dirty_pages, MEMTXATTRS_UNSPECIFIED);
> + stl_le_pci_dma(dev,
> + buf_addr + offsetof(struct igb_mig_dirty_query, status),
> + valid ? IGB_MIG_DIRTY_STATUS_COMPLETE : 0,
> + MEMTXATTRS_UNSPECIFIED);
> +
> + trace_igbvf_mig_dirty_query(s->vfn, (uint64_t)out_size, dirty_pages);
> +}
> +
> +static void igbvf_mig_dirty_ctrl(IgbVfState *s, uint32_t val)
> +{
> + IgbVfMigState *ms = &s->mig;
> + uint64_t pgsize = ms->mig_dirty_pgsize
> + ? ms->mig_dirty_pgsize : IGB_MIG_DIRTY_DEFAULT_PGSIZE;
> +
> + switch (val) {
> + case IGB_MIG_DIRTY_CTRL_ENABLE:
> + ms->mig_dirty_status = igb_core_vf_dirty_enable(s, pgsize,
> + ms->mig_dirty_range_iova,
> + ms->mig_dirty_range_size);
> + if (ms->mig_dirty_status) {
> + break;
> + }
> + trace_igbvf_mig_dirty_enable(s->vfn, pgsize,
> + ms->mig_dirty_range_size / pgsize);
> + break;
> + case IGB_MIG_DIRTY_CTRL_DISABLE:
> + igb_core_vf_dirty_disable(s);
> + ms->mig_dirty_status = IGB_MIG_DIRTY_STATUS_OK;
> + trace_igbvf_mig_dirty_disable(s->vfn);
> + break;
> + case IGB_MIG_DIRTY_CTRL_QUERY:
> + if (!igb_core_vf_dirty_enabled(s)) {
> + ms->mig_dirty_status = IGB_MIG_DIRTY_STATUS_NOT_ENABLED;
> + break;
> + }
> + if (!ms->mig_dirty_buf_addr) {
> + ms->mig_dirty_status = IGB_MIG_DIRTY_STATUS_NO_BUFFER;
> + break;
> + }
> + igbvf_mig_dirty_query(s, pgsize);
> + ms->mig_dirty_status = IGB_MIG_DIRTY_STATUS_OK;
> + break;
> + }
> +}
> +
> static void igbvf_mig_write(void *opaque, hwaddr addr, uint64_t val,
> unsigned size)
> {
> @@ -599,6 +924,31 @@ static void igbvf_mig_write(void *opaque, hwaddr addr, uint64_t val,
> ms->mig_data_buf_addr =
> deposit64(ms->mig_data_buf_addr, 32, 32, val);
> break;
> + case IGB_MIG_DIRTY_PGSIZE:
> + ms->mig_dirty_pgsize = (uint32_t)val;
> + break;
> + case IGB_MIG_DIRTY_CTRL:
> + igbvf_mig_dirty_ctrl(s, (uint32_t)val);
> + break;
> + case IGB_MIG_DIRTY_RANGE_IOVA_LO:
> + ms->mig_dirty_range_iova =
> + deposit64(ms->mig_dirty_range_iova, 0, 32, val);
> + break;
> + case IGB_MIG_DIRTY_RANGE_IOVA_HI:
> + ms->mig_dirty_range_iova =
> + deposit64(ms->mig_dirty_range_iova, 32, 32, val);
> + break;
> + case IGB_MIG_DIRTY_RANGE_SIZE:
> + ms->mig_dirty_range_size = (uint32_t)val;
DIRTY_RANGE_SIZE is only 32 bits. The companion driver likewise passes a
u64 size to a u32 MMIO helper at the range-programming call. Ranges of
4 GiB or larger silently truncate.
Regards,
Akihiko Odaki
> + break;
> + case IGB_MIG_DIRTY_BUF_ADDR_LO:
> + ms->mig_dirty_buf_addr =
> + deposit64(ms->mig_dirty_buf_addr, 0, 32, val);
> + break;
> + case IGB_MIG_DIRTY_BUF_ADDR_HI:
> + ms->mig_dirty_buf_addr =
> + deposit64(ms->mig_dirty_buf_addr, 32, 32, val);
> + break;
> default:
> qemu_log_mask(LOG_GUEST_ERROR,
> "igbvf: VF%u bad migration BAR write at 0x%"
> @@ -643,5 +993,12 @@ void igbvf_mig_state_reset(IgbVfState *s)
> ms->mig_data_size = igb_core_vf_max_data_size(s);
> ms->mig_data_buf_addr = 0;
> memset(ms->mig_data, 0, sizeof(ms->mig_data));
> +
> + igb_core_vf_dirty_disable(s);
> + ms->mig_dirty_pgsize = 0;
> + ms->mig_dirty_range_iova = 0;
> + ms->mig_dirty_range_size = 0;
> + ms->mig_dirty_buf_addr = 0;
> + ms->mig_dirty_status = IGB_MIG_DIRTY_STATUS_OK;
> trace_igbvf_mig_reset(s->vfn);
> }
> diff --git a/hw/net/igbvf.c b/hw/net/igbvf.c
> index e9f9fc3369d8..398306d3bfc6 100644
> --- a/hw/net/igbvf.c
> +++ b/hw/net/igbvf.c
> @@ -306,6 +306,10 @@ static void igbvf_pci_uninit(PCIDevice *dev)
> {
> IgbVfState *s = IGBVF(dev);
>
> + if (s->mig.migration_cap) {
> + igb_core_vf_dirty_disable(s);
> + }
> +
> pcie_aer_exit(dev);
> pcie_cap_exit(dev);
> msix_unuse_all_vectors(dev);
> diff --git a/hw/net/trace-events b/hw/net/trace-events
> index 0b13a99b3f32..95fdea89d49c 100644
> --- a/hw/net/trace-events
> +++ b/hw/net/trace-events
> @@ -303,6 +303,13 @@ igbvf_mig_set_state_err(uint16_t vfn, uint32_t old_state, uint32_t new_state) "V
> igbvf_mig_save_state(uint16_t vfn, uint32_t size) "VF%u: saved %u bytes of device state"
> igbvf_mig_load_state(uint16_t vfn, uint32_t size) "VF%u: loaded %u bytes of device state"
> igbvf_mig_reset(uint16_t vfn) "VF%u: migration state reset"
> +igbvf_mig_dirty_enable(uint16_t vfn, uint64_t pgsize, uint64_t nbits) "VF%u: dirty tracking enabled pgsize=%"PRIu64" nbits=%"PRIu64
> +igbvf_mig_dirty_disable(uint16_t vfn) "VF%u: dirty tracking disabled"
> +igbvf_mig_dirty_query(uint16_t vfn, uint64_t size, uint32_t dirty_pages) "VF%u: dirty query returned %"PRIu64" bytes, %u dirty pages"
> +
> +# igb_core.c - VF migration diagnostics
> +igb_core_dirty_track_dma(int vfn, uint64_t addr, uint64_t len) "VF%d: dirty DMA addr=0x%"PRIx64" len=%"PRIu64
> +igb_core_dirty_track_dma_drop(int vfn, uint64_t addr, uint64_t len) "VF%d: dirty DMA dropped addr=0x%"PRIx64" len=%"PRIu64" no matching range"
>
> # spapr_llan.c
> spapr_vlan_get_rx_bd_from_pool_found(int pool, int32_t count, uint32_t rx_bufs) "pool=%d count=%"PRId32" rxbufs=%"PRIu32
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH 07/11] igb: Quiesce VFs on STOP and include PF enable state in migration blob
2026-07-27 5:39 ` [RFC PATCH 07/11] igb: Quiesce VFs on STOP and include PF enable state in migration blob Cédric Le Goater
@ 2026-07-27 8:19 ` Akihiko Odaki
0 siblings, 0 replies; 26+ messages in thread
From: Akihiko Odaki @ 2026-07-27 8:19 UTC (permalink / raw)
To: Cédric Le Goater, qemu-devel
Cc: Sriram Yagnaraman, Jason Wang, Alex Williamson,
Michael S . Tsirkin, Peter Xu, Avihai Horon
On 2026/07/27 14:39, Cédric Le Goater wrote:
> Quiesce VF queues when entering the STOP state by disabling RX and TX
> at the PF level (VFRE/VFTE). This ensures no DMA activity occurs while
> the device state is being serialized. The unquiesce path re-enables
> queues when returning to RUNNING.
>
> The VFRE and VFTE registers are shared PF-level registers with one bit
> per VF controlling RX and TX enable respectively. Writing the full
> 32-bit register during restore would clobber other VFs' bits, so these
> bits cannot be included in the regular per-VF register list. Instead,
> save the per-VF VFRE/VFTE bit state at quiesce time - before the bits
> are cleared - and append it to the migration blob as two bytes. On the
> destination, extract these bits and restore them during unquiesce.
>
> For QEMU's emulated igb, the device state machine guarantees no DMA in
> these states. This register establishes the protocol for real hardware
> implementations where DMA drain has latency.
>
> Assisted-by: Claude
> Signed-off-by: Cédric Le Goater <clg@redhat.com>
> ---
> hw/net/igb_migration.h | 4 ++
> hw/net/igb_migration.c | 110 +++++++++++++++++++++++++++++++++++++++--
> hw/net/trace-events | 6 ++-
> 3 files changed, 115 insertions(+), 5 deletions(-)
>
> diff --git a/hw/net/igb_migration.h b/hw/net/igb_migration.h
> index a2e7b363cb04..414412f6392e 100644
> --- a/hw/net/igb_migration.h
> +++ b/hw/net/igb_migration.h
> @@ -75,6 +75,7 @@
> /* MIG_STATUS bits */
> #define IGB_MIG_STATUS_DATA_AVAIL (1u << 0)
> #define IGB_MIG_STATUS_ERROR (1u << 1)
> +#define IGB_MIG_STATUS_QUIESCED (1u << 2)
>
> /* MIG_STATUS error codes in bits [15:8], valid when ERROR bit is set */
> #define IGB_MIG_STATUS_ERR_SHIFT 8
> @@ -143,6 +144,9 @@ typedef struct IgbVfMigState {
> uint32_t mig_data_size;
> uint64_t mig_data_buf_addr;
>
> + bool mig_saved_vfre;
> + bool mig_saved_vfte;
> +
> uint32_t mig_dirty_pgsize;
> uint64_t mig_dirty_range_iova;
> uint32_t mig_dirty_range_size;
> diff --git a/hw/net/igb_migration.c b/hw/net/igb_migration.c
> index 9a9ce902bb4a..187077c14c48 100644
> --- a/hw/net/igb_migration.c
> +++ b/hw/net/igb_migration.c
> @@ -95,6 +95,7 @@ static IGBCore *igbvf_get_core(IgbVfState *s)
> * { uint32_t offset; uint32_t value; } regs[num_regs]
> * uint32_t num_tx_ctx (number of TX queue context blocks)
> * { raw struct igb_tx data } tx_ctx[num_tx_ctx]
> + * uint32_t vfre_vfte (VFRE in [7:0], VFTE in [15:8])
> */
>
> /* Maximum number of registers in the VF state slice */
> @@ -227,6 +228,13 @@ static uint32_t *igb_core_vf_save_ra(IGBCore *core, uint16_t vfn,
> return p;
> }
>
> +static uint32_t *igb_core_vf_save_vfre_vfte(uint32_t *p,
> + bool vfre, bool vfte)
> +{
> + *p++ = cpu_to_le32((uint32_t)vfre | ((uint32_t)vfte << 8));
> + return p;
> +}
> +
> static uint32_t *igb_core_vf_save_tx_ctx(IGBCore *core, int queue,
> uint32_t *p)
> {
> @@ -243,12 +251,14 @@ static size_t igb_core_vf_state_max_size(int num_fixed_regs)
> + num_fixed_regs * 2 * sizeof(uint32_t) /* fixed reg pairs */
> + max_ra_regs * 2 * sizeof(uint32_t) /* RA reg pairs */
> + sizeof(uint32_t) /* num_tx_ctx */
> - + 2 * sizeof(struct igb_tx); /* TX context */
> + + 2 * sizeof(struct igb_tx) /* TX context */
> + + sizeof(uint32_t); /* VFRE + VFTE */
> }
>
> static int igb_core_vf_save_state(IgbVfState *s,
> void *buf, size_t buf_size)
> {
> + IgbVfMigState *ms = &s->mig;
> IGBCore *core = igbvf_get_core(s);
> uint32_t offsets[IGB_VF_MAX_REGS];
> int num_regs, total_regs;
> @@ -296,9 +306,15 @@ static int igb_core_vf_save_state(IgbVfState *s,
> p = igb_core_vf_save_tx_ctx(core, q0, p);
> p = igb_core_vf_save_tx_ctx(core, q1, p);
>
> + /* Per-VF VFRE/VFTE enable bits */
> + p = igb_core_vf_save_vfre_vfte(p, ms->mig_saved_vfre,
> + ms->mig_saved_vfte);
> +
> size = (uint8_t *)p - (uint8_t *)buf;
>
> - trace_igbvf_mig_save_state(s->vfn, size);
> + trace_igbvf_mig_save_state(s->vfn, size, ms->mig_saved_vfre,
> + ms->mig_saved_vfte,
> + core->mac[VFRE]);
> return size;
> }
>
> @@ -310,6 +326,16 @@ static int igb_core_vf_max_data_size(IgbVfState *s)
> return size;
> }
>
> +static const void *igb_core_vf_load_vfre_vfte(const void *p,
> + bool *out_vfre, bool *out_vfte)
> +{
> + uint32_t val = le32_to_cpu(*(const uint32_t *)p);
> +
> + *out_vfre = !!(val & 0xff);
> + *out_vfte = !!((val >> 8) & 0xff);
> + return (const uint8_t *)p + sizeof(uint32_t);
> +}
> +
> static const void *igb_core_vf_load_tx_ctx(IGBCore *core, int queue,
> const void *data)
> {
> @@ -323,6 +349,7 @@ static const void *igb_core_vf_load_tx_ctx(IGBCore *core, int queue,
> static int igb_core_vf_load_state(IgbVfState *s,
> const void *buf, size_t size)
> {
> + IgbVfMigState *ms = &s->mig;
> IGBCore *core = igbvf_get_core(s);
> const uint32_t *p = buf;
> uint32_t magic, version, saved_vfn, num_regs, num_tx;
> @@ -378,7 +405,13 @@ static int igb_core_vf_load_state(IgbVfState *s,
> * destination-specific IRTE references after migration.
> */
>
> - trace_igbvf_mig_load_state(s->vfn, (uint32_t)size);
> + /* Per-VF VFRE/VFTE enable bits */
> + p = igb_core_vf_load_vfre_vfte(p, &ms->mig_saved_vfre,
> + &ms->mig_saved_vfte);
> +
> + trace_igbvf_mig_load_state(s->vfn, (uint32_t)size,
> + ms->mig_saved_vfre,
> + ms->mig_saved_vfte);
> return 0;
> }
>
> @@ -387,6 +420,12 @@ static int igbvf_mig_load(IgbVfState *s, const void *buf, size_t size)
> IGBCore *core = igbvf_get_core(s);
> int ret;
>
> + /*
> + * Pre-load: Clear the VFLRE bit before restoring state so the PF
> + * watchdog does not overwrite what we are about to load.
> + */
> + core->mac[VFLRE] &= ~BIT(s->vfn);
> +
> ret = igb_core_vf_load_state(s, buf, size);
> if (ret < 0) {
> return ret;
> @@ -556,6 +595,45 @@ static bool igb_core_vf_dirty_query(IgbVfState *s,
> return false;
> }
>
> +/* Quiesce a VF by disabling its RX and TX at the PF level. */
> +static void igb_core_vf_quiesce(IgbVfState *s)
> +{
> + IgbVfMigState *ms = &s->mig;
> + IGBCore *core = igbvf_get_core(s);
> +
> + ms->mig_saved_vfre = !!(core->mac[VFRE] & BIT(s->vfn));
> + ms->mig_saved_vfte = !!(core->mac[VFTE] & BIT(s->vfn));
> +
> + core->mac[VFRE] &= ~BIT(s->vfn);
> + core->mac[VFTE] &= ~BIT(s->vfn);
> + trace_igbvf_mig_quiesce(s->vfn, core->mac[VFRE], core->mac[VFTE]);
> +}
> +
> +static void igb_core_vf_unquiesce(IgbVfState *s)
> +{
> + IgbVfMigState *ms = &s->mig;
> + IGBCore *core = igbvf_get_core(s);
> + bool re = ms->mig_saved_vfre;
> + bool te = ms->mig_saved_vfte;
> +
> + if (re) {
> + core->mac[VFRE] |= BIT(s->vfn);
> + } else {
> + core->mac[VFRE] &= ~BIT(s->vfn);
> + }
> + if (te) {
> + core->mac[VFTE] |= BIT(s->vfn);
> + } else {
> + core->mac[VFTE] &= ~BIT(s->vfn);
> + }
> +
> + trace_igbvf_mig_unquiesce(s->vfn, core->mac[VFRE], core->mac[VFTE]);
> +
> + if (re) {
> + igb_start_recv(core);
> + }
> +}
> +
> /* ================================================================
> * Migration BAR register read/write handlers
> * ================================================================ */
> @@ -599,6 +677,10 @@ static bool igbvf_mig_set_state(IgbVfState *s, uint32_t new_state)
> old == IGB_MIG_STATE_ERROR) {
> igb_core_vf_dirty_disable(s);
> }
> + if (old == IGB_MIG_STATE_RUNNING ||
> + old == IGB_MIG_STATE_PRE_COPY) {
> + igb_core_vf_quiesce(s);
> + }
ERROR → STOP is accepted, but quiescing only occurs from RUNNING or
PRE_COPY. For example, a missing DATA_XFER buffer enters ERROR while
RX/TX remain enabled; moving to STOP then advertises QUIESCED while VF
DMA is still enabled.
Regards,
Akihiko Odaki
> break;
>
> case IGB_MIG_STATE_RUNNING:
> @@ -609,6 +691,9 @@ static bool igbvf_mig_set_state(IgbVfState *s, uint32_t new_state)
> if (old == IGB_MIG_STATE_PRE_COPY) {
> igb_core_vf_dirty_disable(s);
> }
> + if (old == IGB_MIG_STATE_STOP) {
> + igb_core_vf_unquiesce(s);
> + }
> break;
>
> case IGB_MIG_STATE_STOP_COPY:
> @@ -616,6 +701,9 @@ static bool igbvf_mig_set_state(IgbVfState *s, uint32_t new_state)
> old != IGB_MIG_STATE_PRE_COPY) {
> return false;
> }
> + if (old == IGB_MIG_STATE_PRE_COPY) {
> + igb_core_vf_quiesce(s);
> + }
> ret = igb_core_vf_save_state(s, ms->mig_data, sizeof(ms->mig_data));
> if (ret < 0) {
> ms->mig_error = -ret;
> @@ -656,6 +744,20 @@ static uint32_t igbvf_mig_get_status(IgbVfState *s)
> status |= IGB_MIG_STATUS_DATA_AVAIL;
> }
>
> + /*
> + * QUIESCED tells the driver it is safe to read device state.
> + * STOP and STOP_COPY must guarantee that no VF DMA is in flight;
> + * igb_core_vf_quiesce() enforces this by disabling RX/TX for the
> + * VF.
> + *
> + * Under QEMU, DMA completes synchronously within the MMIO handler
> + * so there is nothing to drain, but real hardware would need this.
> + */
> + if (ms->mig_state == IGB_MIG_STATE_STOP ||
> + ms->mig_state == IGB_MIG_STATE_STOP_COPY) {
> + status |= IGB_MIG_STATUS_QUIESCED;
> + }
> +
> return status;
> }
>
> @@ -1000,5 +1102,7 @@ void igbvf_mig_state_reset(IgbVfState *s)
> ms->mig_dirty_range_size = 0;
> ms->mig_dirty_buf_addr = 0;
> ms->mig_dirty_status = IGB_MIG_DIRTY_STATUS_OK;
> + ms->mig_saved_vfre = true;
> + ms->mig_saved_vfte = true;
> trace_igbvf_mig_reset(s->vfn);
> }
> diff --git a/hw/net/trace-events b/hw/net/trace-events
> index 95fdea89d49c..f4940e3e176d 100644
> --- a/hw/net/trace-events
> +++ b/hw/net/trace-events
> @@ -300,8 +300,8 @@ igbvf_mig_bar_read(uint16_t vfn, uint64_t addr, uint64_t val) "VF%u: BAR read ad
> igbvf_mig_bar_write(uint16_t vfn, uint64_t addr, uint64_t val) "VF%u: BAR write addr=0x%"PRIx64" val=0x%"PRIx64
> igbvf_mig_set_state(uint16_t vfn, uint32_t old_state, uint32_t new_state) "VF%u: state %u -> %u"
> igbvf_mig_set_state_err(uint16_t vfn, uint32_t old_state, uint32_t new_state) "VF%u: invalid transition %u -> %u"
> -igbvf_mig_save_state(uint16_t vfn, uint32_t size) "VF%u: saved %u bytes of device state"
> -igbvf_mig_load_state(uint16_t vfn, uint32_t size) "VF%u: loaded %u bytes of device state"
> +igbvf_mig_save_state(uint16_t vfn, int size, bool vfre, bool vfte, uint32_t reg_vfre) "VF%u: saved %d bytes vfre=%d vfte=%d VFRE=0x%x"
> +igbvf_mig_load_state(uint16_t vfn, uint32_t size, bool vfre, bool vfte) "VF%u: loaded %u bytes vfre=%d vfte=%d"
> igbvf_mig_reset(uint16_t vfn) "VF%u: migration state reset"
> igbvf_mig_dirty_enable(uint16_t vfn, uint64_t pgsize, uint64_t nbits) "VF%u: dirty tracking enabled pgsize=%"PRIu64" nbits=%"PRIu64
> igbvf_mig_dirty_disable(uint16_t vfn) "VF%u: dirty tracking disabled"
> @@ -310,6 +310,8 @@ igbvf_mig_dirty_query(uint16_t vfn, uint64_t size, uint32_t dirty_pages) "VF%u:
> # igb_core.c - VF migration diagnostics
> igb_core_dirty_track_dma(int vfn, uint64_t addr, uint64_t len) "VF%d: dirty DMA addr=0x%"PRIx64" len=%"PRIu64
> igb_core_dirty_track_dma_drop(int vfn, uint64_t addr, uint64_t len) "VF%d: dirty DMA dropped addr=0x%"PRIx64" len=%"PRIu64" no matching range"
> +igbvf_mig_quiesce(uint16_t vfn, uint32_t vfre, uint32_t vfte) "VF%u: quiesce VFRE=0x%x VFTE=0x%x"
> +igbvf_mig_unquiesce(uint16_t vfn, uint32_t vfre, uint32_t vfte) "VF%u: unquiesce VFRE=0x%x VFTE=0x%x"
>
> # spapr_llan.c
> spapr_vlan_get_rx_bd_from_pool_found(int pool, int32_t count, uint32_t rx_bufs) "pool=%d count=%"PRId32" rxbufs=%"PRIu32
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH 08/11] igb: Fix post-migration RX ring deadlock
2026-07-27 5:39 ` [RFC PATCH 08/11] igb: Fix post-migration RX ring deadlock Cédric Le Goater
@ 2026-07-27 9:36 ` Akihiko Odaki
0 siblings, 0 replies; 26+ messages in thread
From: Akihiko Odaki @ 2026-07-27 9:36 UTC (permalink / raw)
To: Cédric Le Goater, qemu-devel
Cc: Sriram Yagnaraman, Jason Wang, Alex Williamson,
Michael S . Tsirkin, Peter Xu, Avihai Horon
On 2026/07/27 14:39, Cédric Le Goater wrote:
> After migration, the VF unquiesce path calls igb_start_recv() to kick
> RX processing. This does not trigger an interrupt, so NAPI stays idle
> and the RX ring is never replenished.
>
> Replace igb_start_recv() with igb_core_vf_rearm_irqs(): re-apply the
> VF's PVT shadow register into the PF aggregates, then raise interrupt
> causes via igb_set_eics(). This follows the normal EICR -> EIMS ->
> MSI-X routing. NAPI wakes, replenishes the ring via
> igbvf_alloc_rx_buffers(), and writes RDT - breaking the deadlock.
I suppose NAPI refers to a Linux-internal concept. Please keep the
device emulation OS-neutral.
>
> The PVT re-application is needed because the L1 PF driver's normal
> interrupt handling may have cleared EIMS between state load and
> unquiesce.
>
> Assisted-by: Claude
> Signed-off-by: Cédric Le Goater <clg@redhat.com>
> ---
> hw/net/igb_core.h | 1 +
> hw/net/igb_core.c | 12 ++++++++++++
> hw/net/igb_migration.c | 2 +-
> 3 files changed, 14 insertions(+), 1 deletion(-)
>
> diff --git a/hw/net/igb_core.h b/hw/net/igb_core.h
> index 50cee2c7683c..3db520048732 100644
> --- a/hw/net/igb_core.h
> +++ b/hw/net/igb_core.h
> @@ -147,5 +147,6 @@ igb_start_recv(IGBCore *core);
>
> IGBCore *igb_pf_get_core(void *pf);
> void igb_core_vf_propagate_irqs(IGBCore *core, uint16_t vfn);
> +void igb_core_vf_rearm_irqs(IGBCore *core, uint16_t vfn);
> void igb_core_vf_propagate_ivar(IGBCore *core, uint16_t vfn);
> #endif
> diff --git a/hw/net/igb_core.c b/hw/net/igb_core.c
> index 03c349575c0a..1621ee6602d5 100644
> --- a/hw/net/igb_core.c
> +++ b/hw/net/igb_core.c
> @@ -4589,6 +4589,18 @@ void igb_core_vf_propagate_irqs(IGBCore *core, uint16_t vfn)
> core->mac[EICR] &= ~(0x7 << shift);
> }
>
> +/*
> + * Re-apply VF interrupt enables to PF aggregates and raise interrupt
> + * causes to wake NAPI so it replenishes the ring after migration.
> + */
> +void igb_core_vf_rearm_irqs(IGBCore *core, uint16_t vfn)
> +{
> + uint32_t shift = 22 - vfn * IGBVF_MSIX_VEC_NUM;
> +
> + igb_core_vf_propagate_irqs(core, vfn);
> + igb_set_eics(core, EICS, 0x7 << shift);
This fabricates all three interrupt causes instead of restoring the
actual pending slice.
> +}
> +
> /*
> * Re-apply VTIVAR -> IVAR0 interrupt routing. The L1 PF driver
> * may have overwritten the shared IVAR0 entries with its own
> diff --git a/hw/net/igb_migration.c b/hw/net/igb_migration.c
> index 187077c14c48..a0b4a52c865f 100644
> --- a/hw/net/igb_migration.c
> +++ b/hw/net/igb_migration.c
> @@ -630,7 +630,7 @@ static void igb_core_vf_unquiesce(IgbVfState *s)
> trace_igbvf_mig_unquiesce(s->vfn, core->mac[VFRE], core->mac[VFTE]);
>
> if (re) {
> - igb_start_recv(core);
> + igb_core_vf_rearm_irqs(core, s->vfn);
Interrupt recovery runs only when saved VFRE is set. A valid
RX-disabled/TX-enabled VF can therefore lose a pending TX completion.
Regards,
Akihiko Odaki
> }
> }
>
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH 09/11] igb: Send RARP after VF migration to update bridge FDB
2026-07-27 5:39 ` [RFC PATCH 09/11] igb: Send RARP after VF migration to update bridge FDB Cédric Le Goater
@ 2026-07-27 10:33 ` Akihiko Odaki
0 siblings, 0 replies; 26+ messages in thread
From: Akihiko Odaki @ 2026-07-27 10:33 UTC (permalink / raw)
To: Cédric Le Goater, qemu-devel
Cc: Sriram Yagnaraman, Jason Wang, Alex Williamson,
Michael S . Tsirkin, Peter Xu, Avihai Horon
On 2026/07/27 14:39, Cédric Le Goater wrote:
> After VF migration the L0 bridge FDB (Forwarding Database) still maps
> the VF's MAC to the old tap port, causing a ~30 s connectivity stall
> while peer ARP caches expire.
>
> Add igb_core_vf_get_mac() to look up a VF's MAC address from the PF's
> Receive Address registers, and use it to send a RARP broadcast from
> the PF's network backend during VF unquiesce so the bridge relearns
> the correct port immediately.
The patch message describes the problem this patch intends to solve, but
it does not explain why it should be handled with the device emulation
code. My intuition is that this is a responsibility of a high-level
management layer on L1.
>
> Assisted-by: Claude
> Signed-off-by: Cédric Le Goater <clg@redhat.com>
> ---
> hw/net/igb_core.h | 1 +
> hw/net/igb_core.c | 28 ++++++++++++++++++++++++++++
> hw/net/igb_migration.c | 33 +++++++++++++++++++++++++++++++++
> hw/net/trace-events | 2 ++
> 4 files changed, 64 insertions(+)
>
> diff --git a/hw/net/igb_core.h b/hw/net/igb_core.h
> index 3db520048732..07b4270708c4 100644
> --- a/hw/net/igb_core.h
> +++ b/hw/net/igb_core.h
> @@ -149,4 +149,5 @@ IGBCore *igb_pf_get_core(void *pf);
> void igb_core_vf_propagate_irqs(IGBCore *core, uint16_t vfn);
> void igb_core_vf_rearm_irqs(IGBCore *core, uint16_t vfn);
> void igb_core_vf_propagate_ivar(IGBCore *core, uint16_t vfn);
> +bool igb_core_vf_get_mac(IGBCore *core, uint16_t vfn, uint8_t *mac);
> #endif
> diff --git a/hw/net/igb_core.c b/hw/net/igb_core.c
> index 1621ee6602d5..281f3a474392 100644
> --- a/hw/net/igb_core.c
> +++ b/hw/net/igb_core.c
> @@ -4634,3 +4634,31 @@ void igb_core_vf_propagate_ivar(IGBCore *core, uint16_t vfn)
> ((uint32_t)ent << (8 * (n % 4)));
> }
> }
> +
> +bool igb_core_vf_get_mac(IGBCore *core, uint16_t vfn, uint8_t *mac)
> +{
> + uint32_t vf_pool_bit = E1000_RAH_POOL_1 << vfn;
> + static const struct {
> + uint32_t base;
> + int count;
> + } ra_banks[] = {
> + { RA, 16 },
> + { RA2, 8 },
> + };
> + int i, j;
> +
> + for (i = 0; i < ARRAY_SIZE(ra_banks); i++) {
> + for (j = 0; j < ra_banks[i].count; j++) {
> + uint32_t ral_off = ra_banks[i].base + j * 2;
> + uint32_t rah_off = ra_banks[i].base + j * 2 + 1;
> + uint32_t rah_val = core->mac[rah_off];
> +
> + if ((rah_val & E1000_RAH_AV) && (rah_val & vf_pool_bit)) {
> + stl_le_p(mac, core->mac[ral_off]);
> + stw_le_p(mac + 4, rah_val & 0xffff);
> + return true;
> + }
> + }
> + }
> + return false;
> +}
> diff --git a/hw/net/igb_migration.c b/hw/net/igb_migration.c
> index a0b4a52c865f..fce661f68e3d 100644
> --- a/hw/net/igb_migration.c
> +++ b/hw/net/igb_migration.c
> @@ -609,6 +609,36 @@ static void igb_core_vf_quiesce(IgbVfState *s)
> trace_igbvf_mig_quiesce(s->vfn, core->mac[VFRE], core->mac[VFTE]);
> }
>
> +/*
> + * Send a RARP broadcast so the network bridge relearns which port
> + * carries this VF's MAC after migration.
> + */
> +static void igb_core_vf_send_rarp(IGBCore *core, uint16_t vfn)
> +{
> + uint8_t mac[ETH_ALEN];
> + uint8_t buf[60];
> +
> + if (!igb_core_vf_get_mac(core, vfn, mac)) {
> + trace_igbvf_mig_no_mac(vfn);
> + return;
> + }
> +
> + trace_igbvf_mig_send_rarp(vfn, mac[0], mac[1], mac[2],
> + mac[3], mac[4], mac[5]);
> +
> + memset(buf, 0xff, ETH_ALEN);
> + memcpy(buf + 6, mac, ETH_ALEN);
> + stw_be_p(buf + 12, 0x8035); /* ETH_P_RARP */
> + stw_be_p(buf + 14, 1); /* hw addr space: ethernet */
> + stw_be_p(buf + 16, ETH_P_IP); /* protocol addr space */
> + buf[18] = 6; buf[19] = 4; /* hw/proto addr lengths */
> + stw_be_p(buf + 20, 3); /* opcode: RARP request */
> + memcpy(buf + 22, mac, ETH_ALEN);
> + memset(buf + 28, 0, 32);
The target-hardware address is also zeroed rather than populated like
QEMU’s existing RARP builder.
> +
> + qemu_send_packet_raw(qemu_get_queue(core->owner_nic), buf, sizeof(buf));
RARP is injected directly into the PF backend, bypassing normal VF TX
VLAN insertion. A VLAN-configured VF therefore announces itself
untagged, so the bridge may learn the wrong VLAN or discard it.
Regards,
Akihiko Odaki
> +}
> +
> static void igb_core_vf_unquiesce(IgbVfState *s)
> {
> IgbVfMigState *ms = &s->mig;
> @@ -632,6 +662,9 @@ static void igb_core_vf_unquiesce(IgbVfState *s)
> if (re) {
> igb_core_vf_rearm_irqs(core, s->vfn);
> }
> +
> + /* TODO : RARP should be sent only if resumed */
> + igb_core_vf_send_rarp(core, s->vfn);
> }
>
> /* ================================================================
> diff --git a/hw/net/trace-events b/hw/net/trace-events
> index f4940e3e176d..1e38d9ab3697 100644
> --- a/hw/net/trace-events
> +++ b/hw/net/trace-events
> @@ -312,6 +312,8 @@ igb_core_dirty_track_dma(int vfn, uint64_t addr, uint64_t len) "VF%d: dirty DMA
> igb_core_dirty_track_dma_drop(int vfn, uint64_t addr, uint64_t len) "VF%d: dirty DMA dropped addr=0x%"PRIx64" len=%"PRIu64" no matching range"
> igbvf_mig_quiesce(uint16_t vfn, uint32_t vfre, uint32_t vfte) "VF%u: quiesce VFRE=0x%x VFTE=0x%x"
> igbvf_mig_unquiesce(uint16_t vfn, uint32_t vfre, uint32_t vfte) "VF%u: unquiesce VFRE=0x%x VFTE=0x%x"
> +igbvf_mig_no_mac(uint16_t vfn) "VF%u: no MAC address found, skipping RARP"
> +igbvf_mig_send_rarp(uint16_t vfn, uint8_t b0, uint8_t b1, uint8_t b2, uint8_t b3, uint8_t b4, uint8_t b5) "VF%u: RARP %02x:%02x:%02x:%02x:%02x:%02x"
>
> # spapr_llan.c
> spapr_vlan_get_rx_bd_from_pool_found(int pool, int32_t count, uint32_t rx_bufs) "pool=%d count=%"PRId32" rxbufs=%"PRIu32
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH 10/11] docs: Add igb VF migration testing setup guide
2026-07-27 5:39 ` [RFC PATCH 10/11] docs: Add igb VF migration testing setup guide Cédric Le Goater
@ 2026-07-27 10:54 ` Akihiko Odaki
0 siblings, 0 replies; 26+ messages in thread
From: Akihiko Odaki @ 2026-07-27 10:54 UTC (permalink / raw)
To: Cédric Le Goater, qemu-devel
Cc: Sriram Yagnaraman, Jason Wang, Alex Williamson,
Michael S . Tsirkin, Peter Xu, Avihai Horon
On 2026/07/27 14:39, Cédric Le Goater wrote:
> Document the NetworkManager configuration and network topology needed
> to test igb VF live migration in a nested virtualization environment.
>
> Signed-off-by: Cédric Le Goater <clg@redhat.com>
> ---
> docs/system/devices/igb-migration.rst | 155 ++++++++++++++++++++++++++
> 1 file changed, 155 insertions(+)
>
> diff --git a/docs/system/devices/igb-migration.rst b/docs/system/devices/igb-migration.rst
> index 4a3a93f01dfc..0e34e4b9fc68 100644
> --- a/docs/system/devices/igb-migration.rst
> +++ b/docs/system/devices/igb-migration.rst
> @@ -134,3 +134,158 @@ The driver fills the request fields, issues ``DIRTY_CTRL=QUERY``, and
> polls ``status`` for completion. The device reads the request, writes
> the dirty bitmap and completion fields via DMA, then sets
> ``status = 1``.
> +
> +Testing setup
> +~~~~~~~~~~~~~
> +
> +The target scenario is nested virtualization::
> +
> + L0 QEMU
> + igb PF with x-vf-migration=on
> + └── VFs with migration BAR + vendor cap
> +
> + L1 kernel
> + igb-vfio-pci variant driver
> + translates VFIO migration v2 ioctls → BAR2 MMIO
> +
> + L1 QEMU (stock, unmodified)
> + vfio-pci device model, standard migration fd
> +
> + L2 guest
> + standard igbvf driver, unaware of migration
> +
> +NetworkManager configuration
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +In a nested setup, the L1 VMs (source and destination) have emulated
> +igb PFs connected to the L0 bridge. By default, NetworkManager
> +acquires DHCP leases on those PF interfaces and on any igb VFs created
> +later. This causes the VF MAC address to be learned on the L0 bridge,
> +which can misdirect iperf3 traffic after migration.
> +
> +To prevent this, configure NetworkManager on **both L1 VMs** and on the
> +**L2 guest disk image**.
> +
> +L1 VMs (source and destination)
> +...............................
> +
> +1. Prevent NetworkManager from managing igbvf interfaces:
> +
> +.. code-block:: bash
> +
> + cat > /etc/NetworkManager/conf.d/99-no-igbvf.conf <<EOF
> + [keyfile]
> + unmanaged-devices=driver:igbvf
> + EOF
> +
> +2. Disable IP on the igb PF connections (keep the interfaces UP for
> + bridging, but with no DHCP lease):
> +
> +.. code-block:: bash
> +
> + # Identify the NM connections for the igb PFs (NOT the virtio management NIC)
> + nmcli -t -f NAME,DEVICE connection show
> +
> + # For each igb PF connection:
> + nmcli connection modify "<igb-pf-connection>" ipv4.method disabled ipv6.method disabled
I suspect we also need: nmcli device reapply
> +
> +3. Reload NetworkManager:
> +
> +.. code-block:: bash
> +
> + nmcli general reload
> +
> +L2 guest disk image
> +...................
> +
> +Use ``virt-customize`` to add the igbvf unmanaged config to the guest
> +image (offline, before any test run):
> +
> +.. code-block:: bash
> +
> + virt-customize -a /srv/migration/rhel10.qcow2 \
> + --write /etc/NetworkManager/conf.d/99-no-igbvf.conf:'[keyfile]
> + unmanaged-devices=driver:igbvf'
> +
> +Network diagram
> +^^^^^^^^^^^^^^^
> +
> +The diagram below shows the nested setup where the source and
> +destination hosts are themselves VMs (L1) running on a physical
> +host (L0) that emulates the igb NIC::
> +
> + ┌────────────────────────────────────────────────────────────────────────────┐
> + │ L0: physical host │
> + │ │
> + │ virbr0 192.168.199.1/24 │
> + │ ├── NFS server: /srv/migration │
> + │ └── iperf3 client: iperf3 -c 192.168.199.200 -t 60 -i 1 │
> + │ │ │
> + │ │ L0 virbr0 bridge (192.168.199.0/24) │
> + │ ────┼──────────┬──────────────────┬────────────────────────────── │
> + │ │ │ │ │
> + │ │ ┌────┴────┐ ┌────┴────┐ │
> + │ │ │ virtio │ │ emulated│ │
> + │ │ │c0:ff:ee:│ │ igb PF │ L0 QEMU (vm6) │
> + │ │ │ :00:06 │ │ + igbvf │ tracks DMA dirty pages │
> + │ │ └────┬────┘ └────┬────┘ │
> + │ │ │ │ │
> + │ ┌───┼──────────┼──────────────────┼──────────────────────────────────┐ │
> + │ │ │ L1: vm6 (source) │ │ │
> + │ │ │ enp1s0: 192.168.199.6 │ │ │
> + │ │ │ (management) │ │ │
> + │ │ │ enp8s0 (igb PF, no IP) │ │
> + │ │ │ │ │ │
> + │ │ │ igb VF0 ──► igb-vfio-pci (VFIO) │ │
> + │ │ │ │ dirty_sync → L0 igbvf │ │
> + │ │ │ │ │ │
> + │ │ │ virbr0 │ VFIO passthrough │ │
> + │ │ │ 192.168.200.1/24 │ │ │
> + │ │ │ │ │ │ │
> + │ │ │ ┌────┼─────────────────────┼───────────────────────────┐ │ │
> + │ │ │ │ │ L2: rhel10 guest │ │ │ │
> + │ │ │ │ │ │ │ │ │
> + │ │ │ │ virtio NIC igb VF (enp7s0) │ │ │
> + │ │ │ │ 192.168.200.130/24 192.168.199.200/24 │ │ │
> + │ │ │ │ (SSH login) (iperf3 data path) │ │ │
> + │ │ │ │ │ │ │ │
> + │ │ │ │ iperf3 -s -D │ (listens on 0.0.0.0) │ │ │
> + │ │ │ └──────────────────────────┼───────────────────────────┘ │ │
> + │ │ │ │ │ │
> + │ │ │ virsh migrate --live ──────┼──────────────────► vm7 │ │
> + │ │ │ │ │ │
> + │ └───┼─────────────────────────────┼──────────────────────────────────┘ │
> + │ │ │ │
> + │ │ iperf3 traffic │ │
> + │ └─────────────────────────────┘ │
> + │ │
> + │ ──────────────────────────────────────────────────────────────── │
> + │ │ │ │
> + │ │ ┌────────┐ │ ┌─────────┐ │
> + │ │ │ virtio │ │ │emulated │ L0 QEMU (vm7) │
> + │ │ │c0:ff:ee│ │ │ igb PF │ │
> + │ │ │ :00:07 │ │ │ + igbvf │ │
> + │ │ └────┬───┘ │ └────┬────┘ │
> + │ ┌──────────────┼───────┼────────┼────────────────────────────────────┐ │
> + │ │ L1: vm7 (destination) │ │ │
> + │ │ enp1s0: 192.168.199.7 │ │ │
> + │ │ (management) enp8s0 (igb PF, no IP) │ │
> + │ │ │ │ │
> + │ │ igb VF0 ──► igb-vfio-pci (VFIO) │ │
> + │ │ │ │ │
> + │ │ virbr0 │ VFIO passthrough │ │
> + │ │ 192.168.200.1/24 │ │ │
> + │ │ │ │ │ │
> + │ │ ┌────┼───────────────────────┼────────────────────────────┐ │ │
> + │ │ │ │ L2: rhel10 (after migration) │ │ │
> + │ │ │ │ │ │ │ │
> + │ │ │ virtio NIC igb VF (enp7s0) │ │ │
> + │ │ │ 192.168.200.130/24 192.168.199.200/24 │ │ │
> + │ │ │ │ │ │ │
> + │ │ │ iperf3 -s -D │ (connection survives) │ │ │
> + │ │ └────────────────────────────┼────────────────────────────┘ │ │
> + │ └───────────────────────────────┼────────────────────────────────────┘ │
> + │ │ │
> + │ iperf3 traffic resumes ─────┘ │
> + │ (same IP, same MAC, same L2 segment → transparent to client) │
> + └────────────────────────────────────────────────────────────────────────────┘
The diagram looks corrupted to me.
Regards,
Akihiko Odaki
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH 11/11] igb: Add migration statistics registers to VF migration BAR
2026-07-27 5:39 ` [RFC PATCH 11/11] igb: Add migration statistics registers to VF migration BAR Cédric Le Goater
@ 2026-07-27 11:05 ` Akihiko Odaki
0 siblings, 0 replies; 26+ messages in thread
From: Akihiko Odaki @ 2026-07-27 11:05 UTC (permalink / raw)
To: Cédric Le Goater, qemu-devel
Cc: Sriram Yagnaraman, Jason Wang, Alex Williamson,
Michael S . Tsirkin, Peter Xu, Avihai Horon
On 2026/07/27 14:39, Cédric Le Goater wrote:
> Add read-only statistics registers to the migration BAR at 0x100-0x11C
> for monitoring dirty page tracking and DMA activity per VF.
>
> The dma_writes counter is also reported in the dirty query buffer so
> the driver gets it alongside the bitmap without an extra MMIO read.
>
> Counters are reset on first DIRTY_CTRL=ENABLE or device reset, so the
> driver can read final values after DIRTY_CTRL=DISABLE.
>
> Signed-off-by: Cédric Le Goater <clg@redhat.com>
> ---
> docs/system/devices/igb-migration.rst | 24 ++++++++++-
> hw/net/igb_core.h | 1 +
> hw/net/igb_migration.h | 22 +++++++++-
> hw/net/igb_migration.c | 59 +++++++++++++++++++++++++--
> hw/net/trace-events | 2 +-
> 5 files changed, 102 insertions(+), 6 deletions(-)
>
> diff --git a/docs/system/devices/igb-migration.rst b/docs/system/devices/igb-migration.rst
> index 0e34e4b9fc68..3f861eed59cd 100644
> --- a/docs/system/devices/igb-migration.rst
> +++ b/docs/system/devices/igb-migration.rst
> @@ -127,7 +127,9 @@ device-written completion fields::
> 0x40 status device 0 = pending, 1 = complete
> 0x44 bitmap_size device Bytes written to bitmap
> 0x48 dirty_page_count device Number of set bits in bitmap
> - 0x4C reserved[12] - Pad to 64-byte cache line
> + 0x4C reserved - Padding
> + 0x50 dma_writes device Total DMA writes since enable
> + 0x58 reserved[10] - Pad to 64-byte cache line
> 0x80 bitmap[] device Dirty page bitmap
>
> The driver fills the request fields, issues ``DIRTY_CTRL=QUERY``, and
> @@ -135,6 +137,26 @@ polls ``status`` for completion. The device reads the request, writes
> the dirty bitmap and completion fields via DMA, then sets
> ``status = 1``.
>
> +Migration statistics
> +~~~~~~~~~~~~~~~~~~~~
> +
> +The migration BAR includes read-only statistics counters in a dedicated
> +aperture at offset ``0x100``. These counters provide a live
> +per-migration-cycle view of activity without requiring a
> +``DIRTY_CTRL=QUERY``::
> +
> + Offset Name Description
> + 0x100 MIG_STAT_DMA_WRITES Total DMA write operations tracked
> + 0x104 MIG_STAT_DMA_BYTES_LO Total DMA bytes written (low 32 bits)
> + 0x108 MIG_STAT_DMA_BYTES_HI Total DMA bytes written (high 32 bits)
> + 0x10C MIG_STAT_DIRTY_PAGES_SET Dirty pages marked since enable
> + 0x110 MIG_STAT_DIRTY_PAGES_CLR Dirty pages cleared by queries
> + 0x114 MIG_STAT_DIRTY_PAGE_COUNT Current dirty pages (set minus cleared)
> + 0x118 MIG_STAT_DIRTY_QUERY_CNT Number of QUERY operations
> +
> +All counters are reset on first ``DIRTY_CTRL=ENABLE`` or device reset,
> +so the driver can read final values after ``DIRTY_CTRL=DISABLE``.
> +
> Testing setup
> ~~~~~~~~~~~~~
>
> diff --git a/hw/net/igb_core.h b/hw/net/igb_core.h
> index 07b4270708c4..91796aec8d7a 100644
> --- a/hw/net/igb_core.h
> +++ b/hw/net/igb_core.h
> @@ -84,6 +84,7 @@ struct IGBCore {
> } tx[IGB_NUM_QUEUES];
>
> IGBVfDirtyState vf_dirty[IGB_MAX_VF_FUNCTIONS];
> + IgbVfMigStats vf_mig_stats[IGB_MAX_VF_FUNCTIONS];
>
> struct NetRxPkt *rx_pkt;
>
> diff --git a/hw/net/igb_migration.h b/hw/net/igb_migration.h
> index 414412f6392e..6d2e3442781f 100644
> --- a/hw/net/igb_migration.h
> +++ b/hw/net/igb_migration.h
> @@ -64,6 +64,15 @@
> #define IGB_MIG_DIRTY_BUF_ADDR_HI 0x038
> #define IGB_MIG_DIRTY_STATUS 0x03C
>
> +/* Migration statistics registers (read-only) */
> +#define IGB_MIG_STAT_DMA_WRITES 0x100
> +#define IGB_MIG_STAT_DMA_BYTES_LO 0x104
> +#define IGB_MIG_STAT_DMA_BYTES_HI 0x108
> +#define IGB_MIG_STAT_DIRTY_PAGES_SET 0x10C
> +#define IGB_MIG_STAT_DIRTY_PAGES_CLR 0x110
> +#define IGB_MIG_STAT_DIRTY_PAGE_COUNT 0x114
> +#define IGB_MIG_STAT_DIRTY_QUERY_CNT 0x118
> +
> /* DEVICE_STATE values - mirrors VFIO migration states */
> #define IGB_MIG_STATE_ERROR 0
> #define IGB_MIG_STATE_STOP 1
> @@ -126,7 +135,9 @@ struct igb_mig_dirty_query {
> uint32_t status;
> uint32_t bitmap_size;
> uint32_t dirty_page_count;
> - uint32_t reserved1[12];
> + uint32_t reserved1;
> + uint64_t dma_writes;
> + uint32_t reserved2[10];
>
> /* Cache line 2+: bitmap (written by device) */
> uint8_t bitmap[];
> @@ -154,6 +165,15 @@ typedef struct IgbVfMigState {
> uint32_t mig_dirty_status;
> } IgbVfMigState;
>
> +typedef struct IgbVfMigStats {
> + uint32_t dma_writes;
dma_writes is exposed and DMA-written as 64 bits, but its backing
counter is only 32 bits. It silently wraps and its upper half is always
zero.
> + uint64_t dma_bytes;
> + uint32_t dirty_pages_set;
> + uint32_t dirty_pages_cleared;
> + uint32_t dirty_page_count;
> + uint32_t dirty_query_count;
> +} IgbVfMigStats;
> +
> typedef struct IgbVfState IgbVfState;
> void igb_pf_init_migration_bar(PCIDevice *dev);
> bool igbvf_add_migration_cap(PCIDevice *dev, Error **errp);
> diff --git a/hw/net/igb_migration.c b/hw/net/igb_migration.c
> index fce661f68e3d..7fce8d535909 100644
> --- a/hw/net/igb_migration.c
> +++ b/hw/net/igb_migration.c
> @@ -458,6 +458,7 @@ void igb_core_dirty_track_dma(IGBCore *core, int vfn,
> dma_addr_t addr, dma_addr_t len)
> {
> IGBVfDirtyState *ds = &core->vf_dirty[vfn];
> + IgbVfMigStats *stats = &core->vf_mig_stats[vfn];
> bool matched = false;
> uint32_t i;
>
> @@ -465,6 +466,8 @@ void igb_core_dirty_track_dma(IGBCore *core, int vfn,
> return;
> }
>
> + stats->dma_writes++;
> + stats->dma_bytes += len;
> trace_igb_core_dirty_track_dma(vfn, addr, len);
>
> for (i = 0; i < ds->num_ranges; i++) {
> @@ -486,7 +489,10 @@ void igb_core_dirty_track_dma(IGBCore *core, int vfn,
>
> for (page = start_page; page <= end_page; page++) {
> if (page < r->nbits) {
> - set_bit(page, r->bitmap);
> + if (!test_and_set_bit(page, r->bitmap)) {
> + stats->dirty_pages_set++;
> + stats->dirty_page_count++;
> + }
> }
> }
> }
> @@ -510,6 +516,15 @@ static uint32_t igb_core_vf_dirty_enable(IgbVfState *s, uint64_t pgsize,
> IGBVfDirtyState *ds = igb_core_vf_dirty_state(s);
> IGBVfDirtyRange *r;
>
> + /*
> + * Reset stats on first enable so the driver can read them after
> + * disable
> + */
> + if (ds->num_ranges == 0) {
> + memset(&igbvf_get_core(s)->vf_mig_stats[s->vfn], 0,
> + sizeof(IgbVfMigStats));
> + }
> +
> if (ds->num_ranges >= IGB_MIG_CAPS_MAX_RANGES) {
> return IGB_MIG_DIRTY_STATUS_TOO_MANY_RANGES;
> }
> @@ -561,7 +576,9 @@ static bool igb_core_vf_dirty_query(IgbVfState *s,
> void *buf, size_t buf_size, size_t *out_size,
> uint64_t range_iova, uint64_t range_size)
> {
> - IGBVfDirtyState *ds = igb_core_vf_dirty_state(s);
> + IGBCore *core = igbvf_get_core(s);
> + IGBVfDirtyState *ds = &core->vf_dirty[s->vfn];
> + IgbVfMigStats *stats = &core->vf_mig_stats[s->vfn];
> uint32_t i;
>
> for (i = 0; i < ds->num_ranges; i++) {
> @@ -587,6 +604,13 @@ static bool igb_core_vf_dirty_query(IgbVfState *s,
> bitmap_clear(r->bitmap, start_page, n);
> }
>
> + {
> + uint32_t cleared = bitmap_count_one(buf, count);
> + stats->dirty_pages_cleared += cleared;
> + stats->dirty_page_count -= MIN(stats->dirty_page_count, cleared);
> + stats->dirty_query_count++;
> + }
> +
> *out_size = bitmap_empty(buf, count) ? 0 : DIV_ROUND_UP(count, 8);
> return true;
> }
> @@ -881,6 +905,7 @@ static void igbvf_mig_data_xfer(IgbVfState *s, uint32_t val)
> static uint64_t igbvf_mig_read(void *opaque, hwaddr addr, unsigned size)
> {
> IgbVfState *s = opaque;
> + IgbVfMigStats *stats = &igbvf_get_core(s)->vf_mig_stats[s->vfn];
> IgbVfMigState *ms = &s->mig;
> uint64_t val = 0;
>
> @@ -915,6 +940,27 @@ static uint64_t igbvf_mig_read(void *opaque, hwaddr addr, unsigned size)
> case IGB_MIG_DIRTY_STATUS:
> val = ms->mig_dirty_status;
> break;
> + case IGB_MIG_STAT_DMA_WRITES:
> + val = stats->dma_writes;
> + break;
> + case IGB_MIG_STAT_DMA_BYTES_LO:
> + val = (uint32_t)stats->dma_bytes;
> + break;
> + case IGB_MIG_STAT_DMA_BYTES_HI:
> + val = (uint32_t)(stats->dma_bytes >> 32);
> + break;
The split live reads of DMA_BYTES_LO/HI can tear across a low-word rollover.
> + case IGB_MIG_STAT_DIRTY_PAGES_SET:
> + val = stats->dirty_pages_set;
> + break;
> + case IGB_MIG_STAT_DIRTY_PAGES_CLR:
> + val = stats->dirty_pages_cleared;
> + break;
> + case IGB_MIG_STAT_DIRTY_PAGE_COUNT:
> + val = stats->dirty_page_count;
> + break;
> + case IGB_MIG_STAT_DIRTY_QUERY_CNT:
> + val = stats->dirty_query_count;
> + break;
> default:
> qemu_log_mask(LOG_GUEST_ERROR,
> "igbvf: VF%u bad migration BAR read at 0x%"
> @@ -942,6 +988,7 @@ static uint32_t igbvf_mig_dirty_count(const void *bitmap, size_t size)
> static void igbvf_mig_dirty_query(IgbVfState *s, uint64_t pgsize)
> {
> IgbVfMigState *ms = &s->mig;
> + IgbVfMigStats *stats = &igbvf_get_core(s)->vf_mig_stats[s->vfn];
> PCIDevice *dev = pcie_sriov_get_pf(PCI_DEVICE(s));
> uint64_t buf_addr = ms->mig_dirty_buf_addr;
> uint64_t range_iova = 0, range_size = 0;
> @@ -982,12 +1029,17 @@ static void igbvf_mig_dirty_query(IgbVfState *s, uint64_t pgsize)
> stl_le_pci_dma(dev,
> buf_addr + offsetof(struct igb_mig_dirty_query, dirty_page_count),
> dirty_pages, MEMTXATTRS_UNSPECIFIED);
> + stq_le_pci_dma(dev,
> + buf_addr + offsetof(struct igb_mig_dirty_query, dma_writes),
> + stats->dma_writes,
> + MEMTXATTRS_UNSPECIFIED);
> stl_le_pci_dma(dev,
> buf_addr + offsetof(struct igb_mig_dirty_query, status),
> valid ? IGB_MIG_DIRTY_STATUS_COMPLETE : 0,
> MEMTXATTRS_UNSPECIFIED);
>
> - trace_igbvf_mig_dirty_query(s->vfn, (uint64_t)out_size, dirty_pages);
> + trace_igbvf_mig_dirty_query(s->vfn, (uint64_t)out_size, dirty_pages,
> + stats->dma_writes);
> }
>
> static void igbvf_mig_dirty_ctrl(IgbVfState *s, uint32_t val)
> @@ -1135,6 +1187,7 @@ void igbvf_mig_state_reset(IgbVfState *s)
> ms->mig_dirty_range_size = 0;
> ms->mig_dirty_buf_addr = 0;
> ms->mig_dirty_status = IGB_MIG_DIRTY_STATUS_OK;
> + memset(&igbvf_get_core(s)->vf_mig_stats[s->vfn], 0, sizeof(IgbVfMigStats));
PF-controlled VF reset and PF CTRL.RST do not reach here.
Regards,
Akihiko Odaki
> ms->mig_saved_vfre = true;
> ms->mig_saved_vfte = true;
> trace_igbvf_mig_reset(s->vfn);
> diff --git a/hw/net/trace-events b/hw/net/trace-events
> index 1e38d9ab3697..76584b43bf19 100644
> --- a/hw/net/trace-events
> +++ b/hw/net/trace-events
> @@ -305,7 +305,7 @@ igbvf_mig_load_state(uint16_t vfn, uint32_t size, bool vfre, bool vfte) "VF%u: l
> igbvf_mig_reset(uint16_t vfn) "VF%u: migration state reset"
> igbvf_mig_dirty_enable(uint16_t vfn, uint64_t pgsize, uint64_t nbits) "VF%u: dirty tracking enabled pgsize=%"PRIu64" nbits=%"PRIu64
> igbvf_mig_dirty_disable(uint16_t vfn) "VF%u: dirty tracking disabled"
> -igbvf_mig_dirty_query(uint16_t vfn, uint64_t size, uint32_t dirty_pages) "VF%u: dirty query returned %"PRIu64" bytes, %u dirty pages"
> +igbvf_mig_dirty_query(uint16_t vfn, uint64_t size, uint32_t dirty_pages, uint64_t dma_writes) "VF%u: dirty query returned %"PRIu64" bytes, %u dirty pages (dma_writes=%"PRIu64")"
>
> # igb_core.c - VF migration diagnostics
> igb_core_dirty_track_dma(int vfn, uint64_t addr, uint64_t len) "VF%d: dirty DMA addr=0x%"PRIx64" len=%"PRIu64
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH 00/11] igb: Add experimental VF live migration support
2026-07-27 5:39 [RFC PATCH 00/11] igb: Add experimental VF live migration support Cédric Le Goater
` (10 preceding siblings ...)
2026-07-27 5:39 ` [RFC PATCH 11/11] igb: Add migration statistics registers to VF migration BAR Cédric Le Goater
@ 2026-07-27 17:35 ` Cédric Le Goater
2026-07-28 5:50 ` Akihiko Odaki
2026-07-27 20:36 ` Alex Williamson
12 siblings, 1 reply; 26+ messages in thread
From: Cédric Le Goater @ 2026-07-27 17:35 UTC (permalink / raw)
To: qemu-devel
Cc: Akihiko Odaki, Sriram Yagnaraman, Jason Wang, Alex Williamson,
Michael S . Tsirkin, Peter Xu, Avihai Horon
Akihiko,
On 7/27/26 07:39, Cédric Le Goater wrote:
> Hello,
>
> Live migration of VFIO-passthrough devices - SR-IOV VFs, vGPUs - is a
> growing requirement, but real hardware with migration support is
> scarce and hard to debug. An emulated device provides a fully
> controlled testbed for developing and validating the entire software
> stack - vfio-pci variant drivers, VFIO core migration v2 framework,
> QEMU, libvirt - and for tuning complex migration policies such as
> downtime convergence. It also serves as an educational reference for
> understanding VFIO migration end-to-end, from device state
> serialization to dirty page tracking.
>
> This series adds an experimental VF live migration interface to the
> emulated igb (82576) device. It enables a vfio-pci variant driver
> (igb-vfio-pci) to migrate VFs using the standard VFIO migration v2
> protocol with stop-copy and pre-copy support.
>
> The target scenario is nested virtualization:
>
> L0 QEMU (these patches)
> igb PF with x-vf-migration=on
> └── VFs with migration BAR + vendor cap
>
> L1 kernel
> igb-vfio-pci variant driver [1]
> translates VFIO migration v2 ioctls → BAR2 MMIO
>
> L1 QEMU (stock, unmodified)
> vfio-pci device model, standard migration fd
>
> L2 guest
> standard igbvf driver, unaware of migration
>
> The L1 QEMU is completely unmodified -- it sees a standard VFIO
> migratable device and uses the normal migration fd path.
>
> * Design
>
> The migration interface is exposed through a hidden 64KB PCI BAR
> (BAR2) on each VF, discovered via a vendor-specific PCI capability
> ("MIGB", PCI_CAP_ID_VNDR). The BAR exposes a register-based state
> machine that mirrors VFIO migration states (RUNNING, STOP, STOP_COPY,
> RESUMING, PRE_COPY).
>
> Device state is serialized as a versioned blob of per-VF register
> (offset, value) pairs covering control, interrupt, RX/TX queue,
> receive address (RA/RA2), etc. plus TX context descriptors and
> VFRE/VFTE enable bits. The blob is transferred via DMA through the PF
> device, since VFIO owns the VF's IOMMU domain and the variant driver
> maps its DMA buffers through the PF.
>
> Dirty page tracking is implemented with per-range bitmaps maintained
> in IGBCore. All VF DMA paths in igb_core.c (TX data, RX data,
> descriptor writeback) are instrumented to record touched pages. The
> variant driver registers tracked IOVA ranges and queries dirty bitmaps
> through a DMA shared buffer, using a single MMIO doorbell
> (DIRTY_CTRL=QUERY) per iteration.
>
> * Caveats
>
> Patch 1 introduces PCI_BASE_ADDRESS_MEM_ALWAYS_ON, a QEMU-internal
> BAR flag that keeps the migration BAR mapped even after VFIO's
> Function Level Reset clears PCI_COMMAND_MEMORY.
>
> The x-vf-migration property is experimental (x- prefix, default off)
> and the migration BAR register interface may change.
>
> The dirty bitmaps are maintained inside the device, which is not
> realistic for discrete NICs without on-chip DRAM.
>
> * Testing
>
> The target scenario is nested virtualization: L0 runs QEMU with an
> igb PF (x-vf-migration=on), L1 runs the igb-vfio-pci variant driver
> and an unmodified QEMU, and L2 runs a standard igbvf driver.
>
> Migration under iperf3 load works correctly: dirty page tracking
> converges (from ~2000 pages per PRE_COPY iteration down to ~280 at
> STOP_COPY), and STOP_COPY stays under 250ms.
>
> * Todo/Ideas
>
> 1. Add migration blocker when x-vf-migration=on (no VMState yet) or
> add VMState support for L0 migration (dirty bitmaps, tracking
> engines, migration BAR registers, stats)
>
> 2. Add PRE_COPY match data validation (magic, version, caps)
>
> 3. Support driver-provided DMA bitmaps per dirty range
> (DIRTY_RANGE_ADDR_LO/HI)
>
> 4. Add QMP/HMP knobs for error injection and capability tuning:
>
> . Dirty rate throttling: artificial delay or rate limit on DMA
> dirty tracking to simulate different convergence scenarios
> . State blob size inflation: pad the migration blob to stress
> large-state transfers and test DATA_SIZE limits
> . Migration phase timing: expose per-VF counters for time spent
> in each state (PRE_COPY duration, STOP_COPY latency)
> . Hot page simulation: mark specific page ranges as always-dirty
> to test worst-case convergence
> . Error injection: force STATUS error codes, fail DMA transfers,
> corrupt state blobs, mask CAPS bits, etc.
>
> 5. Add qtests for migration state machine transitions, dirty page
> tracking ?
>
> * Credits
>
> Alex Williamson suggested the overall approach: a hidden migration BAR
> discovered via a vendor-specific PCI capability, the "vf-migration"
> device property to gate the feature. Thanks for the ever ongoing
> support and valuable discussions throughout these years.
>
> * AI disclaimer
>
> The lack of a migration-capable device has been a recurring pain point
> for VFIO development over the years, and we hope this proposal
> demonstrates the value of having one.
>
> Claude was used to analyze the IGB PF and VF internal state and
> identify the pain points of a working live migration of such devices.
> The generated code served as a starting point but *significant* time
> was then spent cleaning up, reworking, and shaping it into a clear,
> reviewable proposal. As QEMU does not yet accept AI-assisted
> contributions, this series is submitted as an RFC.
>
> Thanks,
>
> C.
>
> [1] https://github.com/legoater/vfio-pci-extras
>
> Cédric Le Goater (11):
> pci: Add PCI_BASE_ADDRESS_MEM_ALWAYS_ON BAR flag
> igb: Add x-vf-migration property and vendor-specific capability for
> IGBVF
> igb: Add migration BAR with state machine
> igb: Add VF state serialization for live migration
> igb: Add VF post-load fixups for live migration
> igb: Add dirty page tracking for IGBVF migration
> igb: Quiesce VFs on STOP and include PF enable state in migration blob
> igb: Fix post-migration RX ring deadlock
> igb: Send RARP after VF migration to update bridge FDB
> docs: Add igb VF migration testing setup guide
> igb: Add migration statistics registers to VF migration BAR
>
> MAINTAINERS | 6 +
> docs/system/device-emulation.rst | 1 +
> docs/system/devices/igb-migration.rst | 313 +++++++
> docs/system/devices/igb.rst | 6 +
> hw/net/igb_common.h | 11 +
> hw/net/igb_core.h | 8 +
> hw/net/igb_migration.h | 201 +++++
> include/hw/pci/pci.h | 6 +
> hw/net/igb.c | 18 +
> hw/net/igb_core.c | 144 ++-
> hw/net/igb_migration.c | 1194 +++++++++++++++++++++++++
> hw/net/igbvf.c | 29 +-
> hw/pci/pci.c | 6 +-
> hw/net/meson.build | 2 +-
> hw/net/trace-events | 20 +
> 15 files changed, 1939 insertions(+), 26 deletions(-)
> create mode 100644 docs/system/devices/igb-migration.rst
> create mode 100644 hw/net/igb_migration.h
> create mode 100644 hw/net/igb_migration.c
>
First thanks for the quick review ! I will recap here :
* bisectability, will improve the first patches.
* state serialization, the blob management is awful. I was expecting
flames. Yes. it needs a rework. I lack a qbuf-style put/get API. The
contents of the blob need a review. Seems feasible.
* relocation between VFs, on my TODO. The code already checks the vfn but
needs tightening and better support.
* interrupts bits: will look into it. hopefully, the model "only" needs
to be more precise. This was a difficult aspect of the igb to understand
and I am still learning.
* dirty tracking, I worked on 2 different interfaces and some of the code
fell through the cracks. DMA failures, bitmap clearing, size validation,
all need improvements. Mostly addressed already, remaining items are code
reorg. should be fine.
* quiesce: looks like a bug in the model. Will check.
* RARP : I got inspiration from the tests. you are right saying it belongs
to the management layer. Keeping it for now as it simplifies testing, but
will move it out.
* Stats: the addition is from yesterday. Needs fixes indeed.
Thanks,
C.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH 00/11] igb: Add experimental VF live migration support
2026-07-27 5:39 [RFC PATCH 00/11] igb: Add experimental VF live migration support Cédric Le Goater
` (11 preceding siblings ...)
2026-07-27 17:35 ` [RFC PATCH 00/11] igb: Add experimental VF live migration support Cédric Le Goater
@ 2026-07-27 20:36 ` Alex Williamson
2026-07-28 8:25 ` Cédric Le Goater
12 siblings, 1 reply; 26+ messages in thread
From: Alex Williamson @ 2026-07-27 20:36 UTC (permalink / raw)
To: Cédric Le Goater
Cc: qemu-devel, Akihiko Odaki, Sriram Yagnaraman, Jason Wang,
Michael S . Tsirkin, Peter Xu, Avihai Horon, alex
On Mon, 27 Jul 2026 07:39:24 +0200
Cédric Le Goater <clg@redhat.com> wrote:
> Hello,
>
> Live migration of VFIO-passthrough devices - SR-IOV VFs, vGPUs - is a
> growing requirement, but real hardware with migration support is
> scarce and hard to debug. An emulated device provides a fully
> controlled testbed for developing and validating the entire software
> stack - vfio-pci variant drivers, VFIO core migration v2 framework,
> QEMU, libvirt - and for tuning complex migration policies such as
> downtime convergence. It also serves as an educational reference for
> understanding VFIO migration end-to-end, from device state
> serialization to dirty page tracking.
>
> This series adds an experimental VF live migration interface to the
> emulated igb (82576) device. It enables a vfio-pci variant driver
> (igb-vfio-pci) to migrate VFs using the standard VFIO migration v2
> protocol with stop-copy and pre-copy support.
>
> The target scenario is nested virtualization:
>
> L0 QEMU (these patches)
> igb PF with x-vf-migration=on
> └── VFs with migration BAR + vendor cap
>
> L1 kernel
> igb-vfio-pci variant driver [1]
> translates VFIO migration v2 ioctls → BAR2 MMIO
>
> L1 QEMU (stock, unmodified)
> vfio-pci device model, standard migration fd
>
> L2 guest
> standard igbvf driver, unaware of migration
>
> The L1 QEMU is completely unmodified -- it sees a standard VFIO
> migratable device and uses the normal migration fd path.
>
> * Design
>
> The migration interface is exposed through a hidden 64KB PCI BAR
> (BAR2) on each VF, discovered via a vendor-specific PCI capability
> ("MIGB", PCI_CAP_ID_VNDR). The BAR exposes a register-based state
> machine that mirrors VFIO migration states (RUNNING, STOP, STOP_COPY,
> RESUMING, PRE_COPY).
I think you're placing the migration BAR on the VF in order to
implement this in a small footprint, QEMU + vfio-pci variant driver,
without PF guest driver changes. A model that better matches real
world hardware might be to put the migration BAR on the PF, segmented
per VF, and then have the PF driver vend those segments out to the VF
drivers. That would remove the BAR always mapped problem, but expands
the footprint to include the PF driver. However, we're not exactly
clean with respect to the PF driver as implemented here when we're
going around the PF driver's back to setup DMA mappings.
Can we take advantage of the fact that this is a virtual device to
avoid all these warts?
For example, do we really need MMIO BAR space for the register set
exposed or can we prune that down to some key registers and doorbells
and move the rest to memory? We can put the vendor capability in
extended config space to give ourselves more room to work with if
necessary. We also don't really need to play by the physical rules for
access, the variant driver in the L1 kernel can allocate contiguous
ranges and write GPAs into config space registers. L0 QEMU can just
write migration data and dirty bitmaps directly to those GPAs,
bypassing any pretense of DMA mapping.
There might be some tricks we can steal from virtio as it seems to
optionally honor things like vIOMMUs as well. Anyway, if we want to
confine the implementation to the virtual VF, avoiding dependencies on
the PF driver, both at the cross-driver API and device DMA state, I
think we can probably lean harder on QEMU being able to push data into
an arbitrary GPA regardless of the IO topology we're exposing. Thanks,
Alex
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH 00/11] igb: Add experimental VF live migration support
2026-07-27 17:35 ` [RFC PATCH 00/11] igb: Add experimental VF live migration support Cédric Le Goater
@ 2026-07-28 5:50 ` Akihiko Odaki
0 siblings, 0 replies; 26+ messages in thread
From: Akihiko Odaki @ 2026-07-28 5:50 UTC (permalink / raw)
To: Cédric Le Goater, qemu-devel
Cc: Sriram Yagnaraman, Jason Wang, Alex Williamson,
Michael S . Tsirkin, Peter Xu, Avihai Horon
On 2026/07/28 2:35, Cédric Le Goater wrote:
> Akihiko,
>
> On 7/27/26 07:39, Cédric Le Goater wrote:
>> Hello,
>>
>> Live migration of VFIO-passthrough devices - SR-IOV VFs, vGPUs - is a
>> growing requirement, but real hardware with migration support is
>> scarce and hard to debug. An emulated device provides a fully
>> controlled testbed for developing and validating the entire software
>> stack - vfio-pci variant drivers, VFIO core migration v2 framework,
>> QEMU, libvirt - and for tuning complex migration policies such as
>> downtime convergence. It also serves as an educational reference for
>> understanding VFIO migration end-to-end, from device state
>> serialization to dirty page tracking.
This is another good example of using QEMU as a SR-IOV testbed.
>>
>> This series adds an experimental VF live migration interface to the
>> emulated igb (82576) device. It enables a vfio-pci variant driver
>> (igb-vfio-pci) to migrate VFs using the standard VFIO migration v2
>> protocol with stop-copy and pre-copy support.
I suggest looking into virtio-net as an alternative to igb. It would
allow avoiding lots of complexities of igb.
>>
>> The target scenario is nested virtualization:
>>
>> L0 QEMU (these patches)
>> igb PF with x-vf-migration=on
>> └── VFs with migration BAR + vendor cap
>>
>> L1 kernel
>> igb-vfio-pci variant driver [1]
>> translates VFIO migration v2 ioctls → BAR2 MMIO
>>
>> L1 QEMU (stock, unmodified)
>> vfio-pci device model, standard migration fd
>>
>> L2 guest
>> standard igbvf driver, unaware of migration
>>
>> The L1 QEMU is completely unmodified -- it sees a standard VFIO
>> migratable device and uses the normal migration fd path.
>>
>> * Design
>>
>> The migration interface is exposed through a hidden 64KB PCI BAR
>> (BAR2) on each VF, discovered via a vendor-specific PCI capability
>> ("MIGB", PCI_CAP_ID_VNDR). The BAR exposes a register-based state
>> machine that mirrors VFIO migration states (RUNNING, STOP, STOP_COPY,
>> RESUMING, PRE_COPY).
>>
>> Device state is serialized as a versioned blob of per-VF register
>> (offset, value) pairs covering control, interrupt, RX/TX queue,
>> receive address (RA/RA2), etc. plus TX context descriptors and
>> VFRE/VFTE enable bits. The blob is transferred via DMA through the PF
>> device, since VFIO owns the VF's IOMMU domain and the variant driver
>> maps its DMA buffers through the PF.
>>
>> Dirty page tracking is implemented with per-range bitmaps maintained
>> in IGBCore. All VF DMA paths in igb_core.c (TX data, RX data,
>> descriptor writeback) are instrumented to record touched pages. The
>> variant driver registers tracked IOVA ranges and queries dirty bitmaps
>> through a DMA shared buffer, using a single MMIO doorbell
>> (DIRTY_CTRL=QUERY) per iteration.
>>
>> * Caveats
>>
>> Patch 1 introduces PCI_BASE_ADDRESS_MEM_ALWAYS_ON, a QEMU-internal
>> BAR flag that keeps the migration BAR mapped even after VFIO's
>> Function Level Reset clears PCI_COMMAND_MEMORY.
>>
>> The x-vf-migration property is experimental (x- prefix, default off)
>> and the migration BAR register interface may change.
>>
>> The dirty bitmaps are maintained inside the device, which is not
>> realistic for discrete NICs without on-chip DRAM.
>>
>> * Testing
>>
>> The target scenario is nested virtualization: L0 runs QEMU with an
>> igb PF (x-vf-migration=on), L1 runs the igb-vfio-pci variant driver
>> and an unmodified QEMU, and L2 runs a standard igbvf driver.
>>
>> Migration under iperf3 load works correctly: dirty page tracking
>> converges (from ~2000 pages per PRE_COPY iteration down to ~280 at
>> STOP_COPY), and STOP_COPY stays under 250ms.
>>
>> * Todo/Ideas
>>
>> 1. Add migration blocker when x-vf-migration=on (no VMState yet) or
>> add VMState support for L0 migration (dirty bitmaps, tracking
>> engines, migration BAR registers, stats)
If you have AI assistance, why don't you just add migration blocker? It
should be just one prompt away and not more troublesome than leaving
this "Todo". ;)
>> 2. Add PRE_COPY match data validation (magic, version, caps)
>> 3. Support driver-provided DMA bitmaps per dirty range
>> (DIRTY_RANGE_ADDR_LO/HI)
>> 4. Add QMP/HMP knobs for error injection and capability tuning:
>> . Dirty rate throttling: artificial delay or rate limit on DMA
>> dirty tracking to simulate different convergence scenarios
>> . State blob size inflation: pad the migration blob to stress
>> large-state transfers and test DATA_SIZE limits
>> . Migration phase timing: expose per-VF counters for time spent
>> in each state (PRE_COPY duration, STOP_COPY latency)
>> . Hot page simulation: mark specific page ranges as always-dirty
>> to test worst-case convergence
>> . Error injection: force STATUS error codes, fail DMA transfers,
>> corrupt state blobs, mask CAPS bits, etc.
>> 5. Add qtests for migration state machine transitions, dirty page
>> tracking ?
>>
>> * Credits
>>
>> Alex Williamson suggested the overall approach: a hidden migration BAR
>> discovered via a vendor-specific PCI capability, the "vf-migration"
>> device property to gate the feature. Thanks for the ever ongoing
>> support and valuable discussions throughout these years.
>>
>> * AI disclaimer
>>
>> The lack of a migration-capable device has been a recurring pain point
>> for VFIO development over the years, and we hope this proposal
>> demonstrates the value of having one.
>>
>> Claude was used to analyze the IGB PF and VF internal state and
>> identify the pain points of a working live migration of such devices.
>> The generated code served as a starting point but *significant* time
>> was then spent cleaning up, reworking, and shaping it into a clear,
>> reviewable proposal. As QEMU does not yet accept AI-assisted
>> contributions, this series is submitted as an RFC.
>>
>> Thanks,
>>
>> C.
>>
>> [1] https://github.com/legoater/vfio-pci-extras
>>
>> Cédric Le Goater (11):
>> pci: Add PCI_BASE_ADDRESS_MEM_ALWAYS_ON BAR flag
>> igb: Add x-vf-migration property and vendor-specific capability for
>> IGBVF
>> igb: Add migration BAR with state machine
>> igb: Add VF state serialization for live migration
>> igb: Add VF post-load fixups for live migration
>> igb: Add dirty page tracking for IGBVF migration
>> igb: Quiesce VFs on STOP and include PF enable state in migration blob
>> igb: Fix post-migration RX ring deadlock
>> igb: Send RARP after VF migration to update bridge FDB
>> docs: Add igb VF migration testing setup guide
>> igb: Add migration statistics registers to VF migration BAR
>>
>> MAINTAINERS | 6 +
>> docs/system/device-emulation.rst | 1 +
>> docs/system/devices/igb-migration.rst | 313 +++++++
>> docs/system/devices/igb.rst | 6 +
>> hw/net/igb_common.h | 11 +
>> hw/net/igb_core.h | 8 +
>> hw/net/igb_migration.h | 201 +++++
>> include/hw/pci/pci.h | 6 +
>> hw/net/igb.c | 18 +
>> hw/net/igb_core.c | 144 ++-
>> hw/net/igb_migration.c | 1194 +++++++++++++++++++++++++
>> hw/net/igbvf.c | 29 +-
>> hw/pci/pci.c | 6 +-
>> hw/net/meson.build | 2 +-
>> hw/net/trace-events | 20 +
>> 15 files changed, 1939 insertions(+), 26 deletions(-)
>> create mode 100644 docs/system/devices/igb-migration.rst
>> create mode 100644 hw/net/igb_migration.h
>> create mode 100644 hw/net/igb_migration.c
>>
>
> First thanks for the quick review ! I will recap here :
>
> * bisectability, will improve the first patches.
> * state serialization, the blob management is awful. I was expecting
> flames. Yes. it needs a rework. I lack a qbuf-style put/get API. The
> contents of the blob need a review. Seems feasible.
> * relocation between VFs, on my TODO. The code already checks the vfn but
> needs tightening and better support.
> * interrupts bits: will look into it. hopefully, the model "only" needs
> to be more precise. This was a difficult aspect of the igb to understand
> and I am still learning.
> * dirty tracking, I worked on 2 different interfaces and some of the code
> fell through the cracks. DMA failures, bitmap clearing, size validation,
> all need improvements. Mostly addressed already, remaining items are
> code
> reorg. should be fine.
> * quiesce: looks like a bug in the model. Will check.
> * RARP : I got inspiration from the tests. you are right saying it belongs
> to the management layer. Keeping it for now as it simplifies testing,
> but
> will move it out.
> * Stats: the addition is from yesterday. Needs fixes indeed.
Thanks for the summary. Well, the "AI disclaimer" states that you spent
significant time on cleanup, but your recap highlights several critical
issues.
Ideally, AI assistance should minimize manual labor while keeping
quality high. Right now, the workflow seems inverted: the AI created the
basic scaffolding, you spent significant time reworking it, yet critical
flaws not mentioned in the "Caveats" and "Todo/Ideas" sections still
remain. I believe we can do better with AI assistance.
In fact, I am part of the Codex for Open Source program and use GPT-5.6
Sol Ultra extensively for reviews. It is quite effective at raising
quality while cutting down manual effort. Most of the comments I
provided came directly from Codex using a simple prompt: "review each
commit." The model somehow possesses a latent understanding of QEMU
requirements, allowing it to flag issues like bisectability breaks and
lack of L0/L1/L2 boundary enforcement.
I expect that iterating with prompts like "review each commit" and "fix
these issues" would automatically resolve most of these problems,
reducing error-prone human intervention. I am not sure if Claude has the
same depth of virtualization-specific knowledge, but even if it doesn't,
you can feed your recap back into the model to let it handle the heavy
lifting.
Moving forward, I see two potential directions for this work:
- Focus on high-level design: Explore alternative architectures like
hacking virtio-net SR-IOV or keeping RARP out of QEMU. You could send
the results as a fresh RFC while openly noting the rough edges. This
allows us to discuss the design architecture without spending massive
effort polishing the implementation details.
- Focus on code quality and advocacy: Burn more tokens to improve the
code quality. Use this series as a concrete case study to demonstrate
that AI assistance can successfully implement complex virtualization
features. This could help shift project policy regarding AI usage and
pave the way for upstreaming this and future AI-assisted patches.
Whichever direction you choose, reframing the narrative in the cover
letter will better align the series with the project's goals.
Regards,
Akihiko Odaki
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH 00/11] igb: Add experimental VF live migration support
2026-07-27 20:36 ` Alex Williamson
@ 2026-07-28 8:25 ` Cédric Le Goater
0 siblings, 0 replies; 26+ messages in thread
From: Cédric Le Goater @ 2026-07-28 8:25 UTC (permalink / raw)
To: Alex Williamson
Cc: qemu-devel, Akihiko Odaki, Sriram Yagnaraman, Jason Wang,
Michael S . Tsirkin, Peter Xu, Avihai Horon
On 7/27/26 22:36, Alex Williamson wrote:
> On Mon, 27 Jul 2026 07:39:24 +0200
> Cédric Le Goater <clg@redhat.com> wrote:
>
>> Hello,
>>
>> Live migration of VFIO-passthrough devices - SR-IOV VFs, vGPUs - is a
>> growing requirement, but real hardware with migration support is
>> scarce and hard to debug. An emulated device provides a fully
>> controlled testbed for developing and validating the entire software
>> stack - vfio-pci variant drivers, VFIO core migration v2 framework,
>> QEMU, libvirt - and for tuning complex migration policies such as
>> downtime convergence. It also serves as an educational reference for
>> understanding VFIO migration end-to-end, from device state
>> serialization to dirty page tracking.
>>
>> This series adds an experimental VF live migration interface to the
>> emulated igb (82576) device. It enables a vfio-pci variant driver
>> (igb-vfio-pci) to migrate VFs using the standard VFIO migration v2
>> protocol with stop-copy and pre-copy support.
>>
>> The target scenario is nested virtualization:
>>
>> L0 QEMU (these patches)
>> igb PF with x-vf-migration=on
>> └── VFs with migration BAR + vendor cap
>>
>> L1 kernel
>> igb-vfio-pci variant driver [1]
>> translates VFIO migration v2 ioctls → BAR2 MMIO
>>
>> L1 QEMU (stock, unmodified)
>> vfio-pci device model, standard migration fd
>>
>> L2 guest
>> standard igbvf driver, unaware of migration
>>
>> The L1 QEMU is completely unmodified -- it sees a standard VFIO
>> migratable device and uses the normal migration fd path.
>>
>> * Design
>>
>> The migration interface is exposed through a hidden 64KB PCI BAR
>> (BAR2) on each VF, discovered via a vendor-specific PCI capability
>> ("MIGB", PCI_CAP_ID_VNDR). The BAR exposes a register-based state
>> machine that mirrors VFIO migration states (RUNNING, STOP, STOP_COPY,
>> RESUMING, PRE_COPY).
>
> I think you're placing the migration BAR on the VF in order to
> implement this in a small footprint, QEMU + vfio-pci variant driver,
> without PF guest driver changes.
yes.
> A model that better matches real
> world hardware might be to put the migration BAR on the PF, segmented
> per VF, and then have the PF driver vend those segments out to the VF
> drivers.
True.
On the migration topic, I saw that the SR-IOV specs had "VF Migration
State Array" feature, which was deprecated.
> That would remove the BAR always mapped problem,
That's the main problem today.
> but expands
> the footprint to include the PF driver. However, we're not exactly
> clean with respect to the PF driver as implemented here when we're
> going around the PF driver's back to setup DMA mappings.
That's not uncommon today. The PDS vfio-pci variant driver setups DMA
mappings in the PF to migrate the VF. But yes, given that the IGB PF
has no idea that a VF could be migrated, it's a bit of a resource hijack.
> Can we take advantage of the fact that this is a virtual device to
> avoid all these warts?
>
> For example, do we really need MMIO BAR space for the register set
> exposed or can we prune that down to some key registers and doorbells
> and move the rest to memory? We can put the vendor capability in
> extended config space to give ourselves more room to work with if
> necessary.
There is plenty of space in the extended config space. The required
register set is relatively small.
> We also don't really need to play by the physical rules for
> access, the variant driver in the L1 kernel can allocate contiguous
> ranges and write GPAs into config space registers.
yes.
> L0 QEMU can just
> write migration data and dirty bitmaps directly to those GPAs,
> bypassing any pretense of DMA mapping.
yes that works.
A previous implementation of this proposal was allocating GPAs.
I then preferred a solution that was more PCI friendly. Anyhow,
it's not a problem to change it again, once we agree on the
HW interface.
> There might be some tricks we can steal from virtio as it seems to
> optionally honor things like vIOMMUs as well.
Yes. That's where the PCI DMAs mapping were interesting. I will
look at it.
> Anyway, if we want to
> confine the implementation to the virtual VF, avoiding dependencies on
> the PF driver, both at the cross-driver API and device DMA state, I
> think we can probably lean harder on QEMU being able to push data into
> an arbitrary GPA regardless of the IO topology we're exposing.
So, the extra PCI BAR is indeed a problem and we should switch to
extended config space to avoid it. I have been exploring several
ideas and this proposal is an hybrid MMIO/shared buffer solution.
Here are my plans for the next.
Reduce the register set to a minimum and use a shared buffer for
all commands. The driver would allocate a single large enough buffer
for the largest command payload (DIRTY_QUERY with bitmap) and write
its GPA into BUF_ADDR once at init. Each operation is: fill buffer,
kick CTRL, poll STATUS.
Registers in extended config space :
0x00 Header Cap ID
0x04 CAPS Features, max_ranges, pgsizes
0x08 CTRL Doorbell (state transitions, dirty ops)
0x0C STATUS Completion + error code
0x10 BUF_ADDR_LO Shared buffer GPA low
0x14 BUF_ADDR_HI Shared buffer GPA high
Commands and payload :
1. SET_STATE
0x00 target_state driver Requested VFIO state
2. SAVE
0x00 data_size device Bytes written (32-bit)
0x04 reserved - May be more reserved bytes to align data
on 32bits
0x08 data[] device State blob
3. LOAD
0x00 data_size driver Bytes to load (32-bit)
0x04 reserved - May be more reserved
0x08 data[] driver State blob
4. DIRTY_ENABLE
0x00 iova driver Range start (64-bit)
0x08 size driver Range size (64-bit)
0x10 pgsize driver Page granularity
0x14 reserved -
5. DIRTY_DISABLE
6. DIRTY_QUERY
Request (driver):
0x00 iova driver Query range start (64-bit)
0x08 size driver Query range size (64-bit)
0x10 pgsize driver Page granularity (for queries)
0x14 reserved -
Response (device):
0x18 bitmap_size device Bytes in bitmap
0x1C dirty_pages device Set bits count
0x20 dma_writes device DMA writes since enable (64-bit)
0x28 reserved -
0x30 bitmap[] device Dirty page bitmap
7. GET_STATS
0x00 dma_writes device (32-bit)
0x04 dma_bytes device (64-bit)
0x0C dirty_pages_set device
0x10 dirty_pages_clr device
0x14 dirty_page_count device
0x18 dirty_query_cnt device
We could add a shared buffer header (not sure this is useful though)
0x00 command driver Operation to perform
0x04 status device Completion + error code
0x08 payload_size both Size of payload following header
0x0C reserved - Alignment
Do we want 'flags' anywhere ?
Thanks,
C.
^ permalink raw reply [flat|nested] 26+ messages in thread
end of thread, other threads:[~2026-07-28 8:26 UTC | newest]
Thread overview: 26+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-07-27 5:39 [RFC PATCH 00/11] igb: Add experimental VF live migration support Cédric Le Goater
2026-07-27 5:39 ` [RFC PATCH 01/11] pci: Add PCI_BASE_ADDRESS_MEM_ALWAYS_ON BAR flag Cédric Le Goater
2026-07-27 5:39 ` [RFC PATCH 02/11] igb: Add x-vf-migration property and vendor-specific capability for IGBVF Cédric Le Goater
2026-07-27 7:13 ` Akihiko Odaki
2026-07-27 5:39 ` [RFC PATCH 03/11] igb: Add migration BAR with state machine Cédric Le Goater
2026-07-27 7:16 ` Akihiko Odaki
2026-07-27 5:39 ` [RFC PATCH 04/11] igb: Add VF state serialization for live migration Cédric Le Goater
2026-07-27 7:31 ` Akihiko Odaki
2026-07-27 5:39 ` [RFC PATCH 05/11] igb: Add VF post-load fixups " Cédric Le Goater
2026-07-27 7:53 ` Akihiko Odaki
2026-07-27 5:39 ` [RFC PATCH 06/11] igb: Add dirty page tracking for IGBVF migration Cédric Le Goater
2026-07-27 8:15 ` Akihiko Odaki
2026-07-27 5:39 ` [RFC PATCH 07/11] igb: Quiesce VFs on STOP and include PF enable state in migration blob Cédric Le Goater
2026-07-27 8:19 ` Akihiko Odaki
2026-07-27 5:39 ` [RFC PATCH 08/11] igb: Fix post-migration RX ring deadlock Cédric Le Goater
2026-07-27 9:36 ` Akihiko Odaki
2026-07-27 5:39 ` [RFC PATCH 09/11] igb: Send RARP after VF migration to update bridge FDB Cédric Le Goater
2026-07-27 10:33 ` Akihiko Odaki
2026-07-27 5:39 ` [RFC PATCH 10/11] docs: Add igb VF migration testing setup guide Cédric Le Goater
2026-07-27 10:54 ` Akihiko Odaki
2026-07-27 5:39 ` [RFC PATCH 11/11] igb: Add migration statistics registers to VF migration BAR Cédric Le Goater
2026-07-27 11:05 ` Akihiko Odaki
2026-07-27 17:35 ` [RFC PATCH 00/11] igb: Add experimental VF live migration support Cédric Le Goater
2026-07-28 5:50 ` Akihiko Odaki
2026-07-27 20:36 ` Alex Williamson
2026-07-28 8:25 ` Cédric Le Goater
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.