[RFC v2 00/14] virtio-net: early VMStateDescription live migration support

[RFC v2 00/14] virtio-net: early VMStateDescription live migration support

[Jonah Palmer]

This RFC explores using VMStateDescription to move virtio-net migration
work out of stop-and-copy. The goal is to reduce guest-visible downtime
by loading as much virtio-net state as possible during the setup phase,
then deciding at stop-and-copy whether the destination can finish with a
minimal resync or needs to fall back to the existing full reload path.

This is a continuation of the earlier RFC that prototyped the idea with
SaveVMHandlers [1]. This version reworks the approach around
VMStateDescription.

At a high level, the series does the following:
- adds an `early-mig` property for virtio-net and enables it by default
  only for newer machine types, while older machine types keep the
  legacy default through compat properties
- adds an early_setup-based VMSD so virtio-net setup work can be moved
  before stop-and-copy
- snapshots state that was sent early, detects relevant mid-migration
  deltas at stop-and-copy, and falls back to the existing full
  virtio-net reload path if any tracked state changed
- if no tracked deltas are found, skips the full reload and only resends
  the minimal state that still must be synchronized at stop-and-copy
- extends the same model to vhost-net by starting most vhost state early
  and deferring backend binding/finalization until stop-and-copy

This RFC series works as follows:
- state loaded early is treated as provisional
- at stop-and-copy, tracked VirtIODevice/VirtIONet state is compared
  against the early snapshot
- if anything relevant changed, migration falls back to the existing
  full virtio-net reload path
- only the no-delta case keeps the shorter stop-and-copy path

For an actual patch series, the following should be implemented:
- handle deltas individually instead of forcing a full reload whenever
  any tracked state changes
- include support for vhost-vDPA

The `early-mig` property is enabled by default only for machine types >=
10.2. Older machine types keep the legacy default off through compat
properties, so existing machine-version behavior is preserved unless the
user overrides the setting explicitly.

For vhost-net, failures while starting vhost early or during the
stop-and-copy quickstart path are treated as non-fatal to migration. In
those cases the destination continues on the userspace virtio-net
datapath, and the normal post-switchover vhost start path may retry
later. If a delta forces a full reload, any early-started vhost
instance is stopped before restart so notifier/backend state is torn
down safely.

Testing framework:
- Local live migration testing compared:
  - baseline: the current VMSD migration flow
  - early VMSD: this RFC series
- Setup:
  - 30 measured runs after 3 warmup runs per mode (baseline vs early)
  - virtio-net with 4 queue pairs
  - guest UDP request/response probe sampled at 10 ms intervals
  - no configuration deltas injected; only early VMSD path used
- Metrics:
  - query-migrate downtime
  - maximum observed probe gap as a guest-visible interruption proxy

Testing results:
- vhost=on (n=30):
  - query-migrate downtime median: 69.0 ms baseline, 60.5 ms early
    (12.3% lower)
  - maximum probe gap median: 309.0 ms baseline, 282.5 ms early
    (8.6% lower)
  - p95: downtime 85 ms baseline vs 86 ms early; probe gap 338 ms in
    both modes

- vhost=off (n=30):
  - `query-migrate` downtime median: 55.0 ms baseline, 49.0 ms early
    (10.9% lower)
  - maximum probe gap median: 283.5 ms baseline, 252.5 ms early
    (10.9% lower)
  - p95: downtime 69 ms baseline vs 81 ms early; probe gap 310.0 ms
    baseline vs 301.0 ms early

Median results improved relative to baseline in both the vhost=on and
vhost=off runs. Higher-percentile results were mixed, especially for
query-migrate downtime in the vhost=off case. That may partly reflect
the relatively small test setup and the fact that these are broader
migration-level or end-to-end measurements rather than explicit
virtio-net device downtime.

A non-RFC submission will have more targeted testing with larger
migration state and more explicit device-focused measurement.

Comments on the overall VMSD structure, migration compatibility
implications, and whether this is the right direction for early
virtio-net device migration would be appreciated.

[1] https://lore.kernel.org/qemu-devel/20250722124127.2497406-1-jonah.palmer@oracle.com/

Jonah Palmer (14):
  machine,virtio-net: add early-mig property
  virtio,virtio-net: add initial early VMSD for setup-phase migration
  virtio,virtio-net: virtio-delta VMSD - VQ state
  virtio-net: detect VirtIODevice status mid-migration change
  virtio-net: detect VirtIODevice config buffer mid-migration change
  virtio-net: detect VirtIONet MAC addr mid-migration change
  virtio-net: detect VirtIONet MAC table mid-migration changes
  virtio-net: detect VirtIONet status mid-migration change
  virtio-net: detect VirtIONet Rx filter mid-migration changes
  virtio-net: detect VirtIONet VLAN filter table changes
  virtio-net: detect VirtIONet guest offload & MQ mid-migration changes
  virtio-net: detect RSS state mid-migration changes
  virtio-net: detect pending Tx work for VQs mid-migration changes
  virtio-net,vhost-net: early migration support for vhost-net

 hw/core/machine.c              |   5 +
 hw/net/vhost_net.c             | 183 ++++++++++++++
 hw/net/virtio-net.c            | 423 +++++++++++++++++++++++++++++++++
 hw/virtio/virtio.c             | 164 ++++++++++++-
 include/hw/virtio/virtio-net.h |  50 ++++
 include/hw/virtio/virtio.h     |  18 ++
 include/net/vhost_net.h        |   9 +
 7 files changed, 851 insertions(+), 1 deletion(-)

-- 
2.51.0

[RFC v2 01/14] machine,virtio-net: add early-mig property

[Jonah Palmer]

Introduce a new 'early-mig' property that enables the early migration
path for virtio-net devices on machine types >= 10.2:

 - virtio-net-device,early-mig=on
 - virtio-net-pci,early-mig=on
 - virtio-net-pci-transitional,early-mig=on
 - virtio-net-pci-non-transitional,early-mig=on
 - virtio-net-ccw,early-mig=on

To preserve compatibility for older machine types (<= 10.1), add compat
overrides in hw_compat_10_1 to keep the legacy default (off).

With this, machine types 10.2 and newer enable early migration by
default while older machine types retain the previous behavior. Users
may still override explicitly via:

 -device virtio-net-pci,early-mig=off
 or
 -global virtio-net-device.early-mig=off

Follow-up patches will implement the actual early migration feature for
virtio-net devices.

Signed-off-by: Jonah Palmer <jonah.palmer@oracle.com>
---
 hw/core/machine.c              | 5 +++++
 hw/net/virtio-net.c            | 1 +
 include/hw/virtio/virtio-net.h | 1 +
 3 files changed, 7 insertions(+)

diff --git a/hw/core/machine.c b/hw/core/machine.c
index 6cf0e2f404..1b6c7db119 100644
--- a/hw/core/machine.c
+++ b/hw/core/machine.c
@@ -45,6 +45,11 @@ const size_t hw_compat_10_2_len = G_N_ELEMENTS(hw_compat_10_2);
 
 GlobalProperty hw_compat_10_1[] = {
     { TYPE_ACPI_GED, "x-has-hest-addr", "false" },
+    { "virtio-net-device", "early-mig", "off" },
+    { "virtio-net-pci", "early-mig", "off" },
+    { "virtio-net-pci-transitional", "early-mig", "off" },
+    { "virtio-net-pci-non-transitional", "early-mig", "off" },
+    { "virtio-net-ccw", "early-mig", "off" },
     { TYPE_VIRTIO_NET, "host_tunnel", "off" },
     { TYPE_VIRTIO_NET, "host_tunnel_csum", "off" },
     { TYPE_VIRTIO_NET, "guest_tunnel", "off" },
diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index 2a5d642a64..12b3456ca2 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -4258,6 +4258,7 @@ static const Property virtio_net_properties[] = {
                       VIRTIO_NET_F_GUEST_USO6, true),
     DEFINE_PROP_BIT64("host_uso", VirtIONet, host_features,
                       VIRTIO_NET_F_HOST_USO, true),
+    DEFINE_PROP_BOOL("early-mig", VirtIONet, early_mig, true),
     DEFINE_PROP_ON_OFF_AUTO_BIT64("hash-ipv4", VirtIONet,
                                   rss_data.specified_hash_types,
                                   VIRTIO_NET_HASH_REPORT_IPv4 - 1,
diff --git a/include/hw/virtio/virtio-net.h b/include/hw/virtio/virtio-net.h
index 371e376428..ddb141fefc 100644
--- a/include/hw/virtio/virtio-net.h
+++ b/include/hw/virtio/virtio-net.h
@@ -230,6 +230,7 @@ struct VirtIONet {
     struct EBPFRSSContext ebpf_rss;
     uint32_t nr_ebpf_rss_fds;
     char **ebpf_rss_fds;
+    bool early_mig;
 };
 
 size_t virtio_net_handle_ctrl_iov(VirtIODevice *vdev,
-- 
2.51.0

Re: [RFC v2 01/14] machine,virtio-net: add early-mig property

[Eugenio Perez Martin]

On Fri, Mar 20, 2026 at 3:20 PM Jonah Palmer <jonah.palmer@oracle.com> wrote:
>
> Introduce a new 'early-mig' property that enables the early migration
> path for virtio-net devices on machine types >= 10.2:
>
>  - virtio-net-device,early-mig=on
>  - virtio-net-pci,early-mig=on
>  - virtio-net-pci-transitional,early-mig=on
>  - virtio-net-pci-non-transitional,early-mig=on
>  - virtio-net-ccw,early-mig=on
>
> To preserve compatibility for older machine types (<= 10.1), add compat
> overrides in hw_compat_10_1 to keep the legacy default (off).
>
> With this, machine types 10.2 and newer enable early migration by
> default while older machine types retain the previous behavior. Users
> may still override explicitly via:
>
>  -device virtio-net-pci,early-mig=off
>  or
>  -global virtio-net-device.early-mig=off
>
> Follow-up patches will implement the actual early migration feature for
> virtio-net devices.
>
> Signed-off-by: Jonah Palmer <jonah.palmer@oracle.com>
> ---
>  hw/core/machine.c              | 5 +++++
>  hw/net/virtio-net.c            | 1 +
>  include/hw/virtio/virtio-net.h | 1 +
>  3 files changed, 7 insertions(+)
>
> diff --git a/hw/core/machine.c b/hw/core/machine.c
> index 6cf0e2f404..1b6c7db119 100644
> --- a/hw/core/machine.c
> +++ b/hw/core/machine.c
> @@ -45,6 +45,11 @@ const size_t hw_compat_10_2_len = G_N_ELEMENTS(hw_compat_10_2);
>
>  GlobalProperty hw_compat_10_1[] = {
>      { TYPE_ACPI_GED, "x-has-hest-addr", "false" },
> +    { "virtio-net-device", "early-mig", "off" },
> +    { "virtio-net-pci", "early-mig", "off" },
> +    { "virtio-net-pci-transitional", "early-mig", "off" },
> +    { "virtio-net-pci-non-transitional", "early-mig", "off" },
> +    { "virtio-net-ccw", "early-mig", "off" },

Is this needed even if we mark the VMState as optional?

The source might not be able to send the device configuration in some
cases. For example, if it cannot shadow CVQ. If the destination knows
this field is optional, the QEMU source version is irrelevant.

>      { TYPE_VIRTIO_NET, "host_tunnel", "off" },
>      { TYPE_VIRTIO_NET, "host_tunnel_csum", "off" },
>      { TYPE_VIRTIO_NET, "guest_tunnel", "off" },
> diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
> index 2a5d642a64..12b3456ca2 100644
> --- a/hw/net/virtio-net.c
> +++ b/hw/net/virtio-net.c
> @@ -4258,6 +4258,7 @@ static const Property virtio_net_properties[] = {
>                        VIRTIO_NET_F_GUEST_USO6, true),
>      DEFINE_PROP_BIT64("host_uso", VirtIONet, host_features,
>                        VIRTIO_NET_F_HOST_USO, true),
> +    DEFINE_PROP_BOOL("early-mig", VirtIONet, early_mig, true),
>      DEFINE_PROP_ON_OFF_AUTO_BIT64("hash-ipv4", VirtIONet,
>                                    rss_data.specified_hash_types,
>                                    VIRTIO_NET_HASH_REPORT_IPv4 - 1,
> diff --git a/include/hw/virtio/virtio-net.h b/include/hw/virtio/virtio-net.h
> index 371e376428..ddb141fefc 100644
> --- a/include/hw/virtio/virtio-net.h
> +++ b/include/hw/virtio/virtio-net.h
> @@ -230,6 +230,7 @@ struct VirtIONet {
>      struct EBPFRSSContext ebpf_rss;
>      uint32_t nr_ebpf_rss_fds;
>      char **ebpf_rss_fds;
> +    bool early_mig;
>  };
>
>  size_t virtio_net_handle_ctrl_iov(VirtIODevice *vdev,
> --
> 2.51.0
>

Re: [RFC v2 01/14] machine,virtio-net: add early-mig property

[Jonah Palmer]

On 3/23/26 6:25 AM, Eugenio Perez Martin wrote:
> On Fri, Mar 20, 2026 at 3:20 PM Jonah Palmer <jonah.palmer@oracle.com> wrote:
>>
>> Introduce a new 'early-mig' property that enables the early migration
>> path for virtio-net devices on machine types >= 10.2:
>>
>>   - virtio-net-device,early-mig=on
>>   - virtio-net-pci,early-mig=on
>>   - virtio-net-pci-transitional,early-mig=on
>>   - virtio-net-pci-non-transitional,early-mig=on
>>   - virtio-net-ccw,early-mig=on
>>
>> To preserve compatibility for older machine types (<= 10.1), add compat
>> overrides in hw_compat_10_1 to keep the legacy default (off).
>>
>> With this, machine types 10.2 and newer enable early migration by
>> default while older machine types retain the previous behavior. Users
>> may still override explicitly via:
>>
>>   -device virtio-net-pci,early-mig=off
>>   or
>>   -global virtio-net-device.early-mig=off
>>
>> Follow-up patches will implement the actual early migration feature for
>> virtio-net devices.
>>
>> Signed-off-by: Jonah Palmer <jonah.palmer@oracle.com>
>> ---
>>   hw/core/machine.c              | 5 +++++
>>   hw/net/virtio-net.c            | 1 +
>>   include/hw/virtio/virtio-net.h | 1 +
>>   3 files changed, 7 insertions(+)
>>
>> diff --git a/hw/core/machine.c b/hw/core/machine.c
>> index 6cf0e2f404..1b6c7db119 100644
>> --- a/hw/core/machine.c
>> +++ b/hw/core/machine.c
>> @@ -45,6 +45,11 @@ const size_t hw_compat_10_2_len = G_N_ELEMENTS(hw_compat_10_2);
>>
>>   GlobalProperty hw_compat_10_1[] = {
>>       { TYPE_ACPI_GED, "x-has-hest-addr", "false" },
>> +    { "virtio-net-device", "early-mig", "off" },
>> +    { "virtio-net-pci", "early-mig", "off" },
>> +    { "virtio-net-pci-transitional", "early-mig", "off" },
>> +    { "virtio-net-pci-non-transitional", "early-mig", "off" },
>> +    { "virtio-net-ccw", "early-mig", "off" },
> 
> Is this needed even if we mark the VMState as optional?
> 
> The source might not be able to send the device configuration in some
> cases. For example, if it cannot shadow CVQ. If the destination knows
> this field is optional, the QEMU source version is irrelevant.
> 

I think it is still needed, yes. This series adds new VMSDs that are 
registered as top-level savevm sections, and, IIUC, a destination guest 
using an older version of Qemu would not be able to ignore these 
"unknown" sections, even if they're optional for the source to send.

The compat property also preserves the pre-10.2 machine-type default, so 
a 10.1 machine type doesn't silently change the migration behavior when 
it's being run on a newer Qemu version.

I do agree though that being unable to shadow some state should be 
handled as a runtime check for early migration (like we do for 
vhost-user), but that's separate from a machine-type compat default.

>>       { TYPE_VIRTIO_NET, "host_tunnel", "off" },
>>       { TYPE_VIRTIO_NET, "host_tunnel_csum", "off" },
>>       { TYPE_VIRTIO_NET, "guest_tunnel", "off" },
>> diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
>> index 2a5d642a64..12b3456ca2 100644
>> --- a/hw/net/virtio-net.c
>> +++ b/hw/net/virtio-net.c
>> @@ -4258,6 +4258,7 @@ static const Property virtio_net_properties[] = {
>>                         VIRTIO_NET_F_GUEST_USO6, true),
>>       DEFINE_PROP_BIT64("host_uso", VirtIONet, host_features,
>>                         VIRTIO_NET_F_HOST_USO, true),
>> +    DEFINE_PROP_BOOL("early-mig", VirtIONet, early_mig, true),
>>       DEFINE_PROP_ON_OFF_AUTO_BIT64("hash-ipv4", VirtIONet,
>>                                     rss_data.specified_hash_types,
>>                                     VIRTIO_NET_HASH_REPORT_IPv4 - 1,
>> diff --git a/include/hw/virtio/virtio-net.h b/include/hw/virtio/virtio-net.h
>> index 371e376428..ddb141fefc 100644
>> --- a/include/hw/virtio/virtio-net.h
>> +++ b/include/hw/virtio/virtio-net.h
>> @@ -230,6 +230,7 @@ struct VirtIONet {
>>       struct EBPFRSSContext ebpf_rss;
>>       uint32_t nr_ebpf_rss_fds;
>>       char **ebpf_rss_fds;
>> +    bool early_mig;
>>   };
>>
>>   size_t virtio_net_handle_ctrl_iov(VirtIODevice *vdev,
>> --
>> 2.51.0
>>
> 

Re: [RFC v2 01/14] machine,virtio-net: add early-mig property

[Eugenio Perez Martin]

On Tue, Mar 24, 2026 at 3:07 PM Jonah Palmer <jonah.palmer@oracle.com> wrote:
>
>
>
> On 3/23/26 6:25 AM, Eugenio Perez Martin wrote:
> > On Fri, Mar 20, 2026 at 3:20 PM Jonah Palmer <jonah.palmer@oracle.com> wrote:
> >>
> >> Introduce a new 'early-mig' property that enables the early migration
> >> path for virtio-net devices on machine types >= 10.2:
> >>
> >>   - virtio-net-device,early-mig=on
> >>   - virtio-net-pci,early-mig=on
> >>   - virtio-net-pci-transitional,early-mig=on
> >>   - virtio-net-pci-non-transitional,early-mig=on
> >>   - virtio-net-ccw,early-mig=on
> >>
> >> To preserve compatibility for older machine types (<= 10.1), add compat
> >> overrides in hw_compat_10_1 to keep the legacy default (off).
> >>
> >> With this, machine types 10.2 and newer enable early migration by
> >> default while older machine types retain the previous behavior. Users
> >> may still override explicitly via:
> >>
> >>   -device virtio-net-pci,early-mig=off
> >>   or
> >>   -global virtio-net-device.early-mig=off
> >>
> >> Follow-up patches will implement the actual early migration feature for
> >> virtio-net devices.
> >>
> >> Signed-off-by: Jonah Palmer <jonah.palmer@oracle.com>
> >> ---
> >>   hw/core/machine.c              | 5 +++++
> >>   hw/net/virtio-net.c            | 1 +
> >>   include/hw/virtio/virtio-net.h | 1 +
> >>   3 files changed, 7 insertions(+)
> >>
> >> diff --git a/hw/core/machine.c b/hw/core/machine.c
> >> index 6cf0e2f404..1b6c7db119 100644
> >> --- a/hw/core/machine.c
> >> +++ b/hw/core/machine.c
> >> @@ -45,6 +45,11 @@ const size_t hw_compat_10_2_len = G_N_ELEMENTS(hw_compat_10_2);
> >>
> >>   GlobalProperty hw_compat_10_1[] = {
> >>       { TYPE_ACPI_GED, "x-has-hest-addr", "false" },
> >> +    { "virtio-net-device", "early-mig", "off" },
> >> +    { "virtio-net-pci", "", "off" },
> >> +    { "virtio-net-pci-transitional", "early-mig", "off" },
> >> +    { "virtio-net-pci-non-transitional", "early-mig", "off" },
> >> +    { "virtio-net-ccw", "early-mig", "off" },
> >
> > Is this needed even if we mark the VMState as optional?
> >
> > The source might not be able to send the device configuration in some
> > cases. For example, if it cannot shadow CVQ. If the destination knows
> > this field is optional, the QEMU source version is irrelevant.
> >
>
> I think it is still needed, yes. This series adds new VMSDs that are
> registered as top-level savevm sections, and, IIUC, a destination guest
> using an older version of Qemu would not be able to ignore these
> "unknown" sections, even if they're optional for the source to send.
>
> The compat property also preserves the pre-10.2 machine-type default, so
> a 10.1 machine type doesn't silently change the migration behavior when
> it's being run on a newer Qemu version.
>

Ok good point. I'm tempted to propose removing the cmdline argument
then, but I see virtio-mem "x-early-migration" also adds it. I don't
think increasing the migration's complexity by adding command-line
cases is worth it, but I guess there is a reason for that. What about
setting it to true by default?

Also, maybe it is worth exploring using the same name for both
properties. At the same time, I remember that x- prefix was
discouraged so... I'll be happy with any option
{x-,}early-mig{,ration}.

Again, I wouldn't object to adding the cmdline parameter, I'm just
trying to make this as easy to maintain in the future as possible :).

> I do agree though that being unable to shadow some state should be
> handled as a runtime check for early migration (like we do for
> vhost-user), but that's separate from a machine-type compat default.
>

Yes, I missed that vhost-user and vhost-vdpa were out of the series at
this patch. It happened with a few comments. I should have gone back
and solved them myself; apologies! If possible, and if you see it
makes sense, you can expand the patch descriptions with the info.

[RFC v2 02/14] virtio, virtio-net: add initial early VMSD for setup-phase migration

[Jonah Palmer via qemu development]

Adds a separate VMStateDescription for virtio-net that uses the
.early_setup feature. With this feature, we can migrate a virtio-net
device's state earlier, before the stop-and-copy phase.

Future patches will utilize this to move control plane operations out of
the stop-and-copy phase to reduce the downtime latency caused by
migrating a virtio-net device.

A VirtIODevMigration migration data structure is also introduced here
for VirtIODevices to help track the current state of a migration.

The early_load member is used to signal that a VirtIODevice is being
loaded early and to not throw an error regarding vring indices.
Inconsistent indices shouldn't be an issue for a device so long as the
final indices are eventually loaded before the device starts.

Signed-off-by: Jonah Palmer <jonah.palmer@oracle.com>
---
 hw/net/virtio-net.c        | 53 ++++++++++++++++++++++++++++++++++++++
 hw/virtio/virtio.c         | 14 +++++++++-
 include/hw/virtio/virtio.h |  9 +++++++
 3 files changed, 75 insertions(+), 1 deletion(-)

diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index 12b3456ca2..ddd6ed6e62 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -3864,6 +3864,37 @@ static bool failover_hide_primary_device(DeviceListener *listener,
     return qatomic_read(&n->failover_primary_hidden);
 }
 
+static int virtio_net_early_pre_load(void *opaque)
+{
+    VirtIONet *n = opaque;
+    VirtIODevice *vdev = VIRTIO_DEVICE(n);
+
+    vdev->migration->early_load = true;
+    return 0;
+}
+
+static int virtio_net_early_post_load(void *opaque, int version_id)
+{
+    VirtIONet *n = opaque;
+    VirtIODevice *vdev = VIRTIO_DEVICE(n);
+
+    vdev->migration->early_load = false;
+    return 0;
+}
+
+static const VMStateDescription vmstate_virtio_net_early = {
+    .name = "virtio-net-early",
+    .minimum_version_id = VIRTIO_NET_VM_VERSION,
+    .version_id = VIRTIO_NET_VM_VERSION,
+    .early_setup = true,
+    .pre_load = virtio_net_early_pre_load,
+    .post_load = virtio_net_early_post_load,
+    .fields = (const VMStateField[]) {
+        VMSTATE_VIRTIO_DEVICE,
+        VMSTATE_END_OF_LIST()
+    },
+};
+
 static void virtio_net_device_realize(DeviceState *dev, Error **errp)
 {
     VirtIODevice *vdev = VIRTIO_DEVICE(dev);
@@ -4046,6 +4077,21 @@ static void virtio_net_device_realize(DeviceState *dev, Error **errp)
             n->rss_data.specified_hash_types.on_bits |
             n->rss_data.specified_hash_types.auto_bits;
     }
+
+    if (n->early_mig) {
+        if (nc->peer && nc->peer->info->type == NET_CLIENT_DRIVER_VHOST_USER) {
+            /*
+             * vhost-user backend is not currently supported for the early
+             * migration path.
+             */
+            n->early_mig = false;
+        } else {
+            vdev->migration = g_new0(VirtIODevMigration, 1);
+            vdev->migration->early_load = false;
+
+            vmstate_register_any(VMSTATE_IF(n), &vmstate_virtio_net_early, n);
+        }
+    }
 }
 
 static void virtio_net_device_unrealize(DeviceState *dev)
@@ -4090,6 +4136,13 @@ static void virtio_net_device_unrealize(DeviceState *dev)
     g_free(n->rss_data.indirections_table);
     net_rx_pkt_uninit(n->rx_pkt);
     virtio_cleanup(vdev);
+
+    if (n->early_mig) {
+        g_free(vdev->migration);
+        vdev->migration = NULL;
+
+        vmstate_unregister(VMSTATE_IF(n), &vmstate_virtio_net_early, n);
+    }
 }
 
 static void virtio_net_reset(VirtIODevice *vdev)
diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
index 8fcf6cfd0b..48de4a430b 100644
--- a/hw/virtio/virtio.c
+++ b/hw/virtio/virtio.c
@@ -3323,6 +3323,7 @@ virtio_load(VirtIODevice *vdev, QEMUFile *f, int version_id)
     int32_t config_len;
     uint32_t num;
     uint32_t features;
+    bool inconsistent_indices;
     BusState *qbus = qdev_get_parent_bus(DEVICE(vdev));
     VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(qbus);
     VirtioDeviceClass *vdc = VIRTIO_DEVICE_GET_CLASS(vdev);
@@ -3460,6 +3461,14 @@ virtio_load(VirtIODevice *vdev, QEMUFile *f, int version_id)
         if (vdev->vq[i].vring.desc) {
             uint16_t nheads;
 
+            /*
+             * Ring indices will be inconsistent for a VMStateDescription
+             * performing an early load. This shouldn't be an issue as the
+             * final indices will get sent later once the source has been
+             * stopped.
+             */
+            inconsistent_indices = vdev->migration && vdev->migration->early_load;
+
             /*
              * VIRTIO-1 devices migrate desc, used, and avail ring addresses so
              * only the region cache needs to be set up.  Legacy devices need
@@ -3481,12 +3490,15 @@ virtio_load(VirtIODevice *vdev, QEMUFile *f, int version_id)
 
             nheads = vring_avail_idx(&vdev->vq[i]) - vdev->vq[i].last_avail_idx;
             /* Check it isn't doing strange things with descriptor numbers. */
-            if (nheads > vdev->vq[i].vring.num) {
+            if (!inconsistent_indices && nheads > vdev->vq[i].vring.num) {
                 virtio_error(vdev, "VQ %d size 0x%x Guest index 0x%x "
                              "inconsistent with Host index 0x%x: delta 0x%x",
                              i, vdev->vq[i].vring.num,
                              vring_avail_idx(&vdev->vq[i]),
                              vdev->vq[i].last_avail_idx, nheads);
+                inconsistent_indices = true;
+            }
+            if (inconsistent_indices) {
                 vdev->vq[i].used_idx = 0;
                 vdev->vq[i].shadow_avail_idx = 0;
                 vdev->vq[i].inuse = 0;
diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
index 6344bd7b68..4c886eb48b 100644
--- a/include/hw/virtio/virtio.h
+++ b/include/hw/virtio/virtio.h
@@ -99,6 +99,14 @@ enum virtio_device_endian {
     VIRTIO_DEVICE_ENDIAN_BIG,
 };
 
+/**
+ * struct VirtIODevMigration - Common VirtIODevice migration structure
+ * @early_load: Flag to indicate an early virtio_load for the device.
+ */
+typedef struct VirtIODevMigration {
+    bool early_load;
+} VirtIODevMigration;
+
 /**
  * struct VirtIODevice - common VirtIO structure
  * @name: name of the device
@@ -168,6 +176,7 @@ struct VirtIODevice
      */
     EventNotifier config_notifier;
     bool device_iotlb_enabled;
+    VirtIODevMigration *migration;
 };
 
 struct VirtioDeviceClass {
-- 
2.51.0

Re: [RFC v2 02/14] virtio,virtio-net: add initial early VMSD for setup-phase migration

[Eugenio Perez Martin]

On Fri, Mar 20, 2026 at 3:25 PM Jonah Palmer <jonah.palmer@oracle.com> wrote:
>
> Adds a separate VMStateDescription for virtio-net that uses the
> .early_setup feature. With this feature, we can migrate a virtio-net
> device's state earlier, before the stop-and-copy phase.
>
> Future patches will utilize this to move control plane operations out of
> the stop-and-copy phase to reduce the downtime latency caused by
> migrating a virtio-net device.
>
> A VirtIODevMigration migration data structure is also introduced here
> for VirtIODevices to help track the current state of a migration.
>
> The early_load member is used to signal that a VirtIODevice is being
> loaded early and to not throw an error regarding vring indices.
> Inconsistent indices shouldn't be an issue for a device so long as the
> final indices are eventually loaded before the device starts.
>
> Signed-off-by: Jonah Palmer <jonah.palmer@oracle.com>
> ---
>  hw/net/virtio-net.c        | 53 ++++++++++++++++++++++++++++++++++++++
>  hw/virtio/virtio.c         | 14 +++++++++-
>  include/hw/virtio/virtio.h |  9 +++++++
>  3 files changed, 75 insertions(+), 1 deletion(-)
>
> diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
> index 12b3456ca2..ddd6ed6e62 100644
> --- a/hw/net/virtio-net.c
> +++ b/hw/net/virtio-net.c
> @@ -3864,6 +3864,37 @@ static bool failover_hide_primary_device(DeviceListener *listener,
>      return qatomic_read(&n->failover_primary_hidden);
>  }
>
> +static int virtio_net_early_pre_load(void *opaque)
> +{
> +    VirtIONet *n = opaque;
> +    VirtIODevice *vdev = VIRTIO_DEVICE(n);
> +
> +    vdev->migration->early_load = true;
> +    return 0;
> +}
> +
> +static int virtio_net_early_post_load(void *opaque, int version_id)
> +{
> +    VirtIONet *n = opaque;
> +    VirtIODevice *vdev = VIRTIO_DEVICE(n);
> +
> +    vdev->migration->early_load = false;
> +    return 0;
> +}
> +
> +static const VMStateDescription vmstate_virtio_net_early = {
> +    .name = "virtio-net-early",
> +    .minimum_version_id = VIRTIO_NET_VM_VERSION,
> +    .version_id = VIRTIO_NET_VM_VERSION,
> +    .early_setup = true,
> +    .pre_load = virtio_net_early_pre_load,
> +    .post_load = virtio_net_early_post_load,
> +    .fields = (const VMStateField[]) {
> +        VMSTATE_VIRTIO_DEVICE,
> +        VMSTATE_END_OF_LIST()
> +    },
> +};
> +
>  static void virtio_net_device_realize(DeviceState *dev, Error **errp)
>  {
>      VirtIODevice *vdev = VIRTIO_DEVICE(dev);
> @@ -4046,6 +4077,21 @@ static void virtio_net_device_realize(DeviceState *dev, Error **errp)
>              n->rss_data.specified_hash_types.on_bits |
>              n->rss_data.specified_hash_types.auto_bits;
>      }
> +
> +    if (n->early_mig) {
> +        if (nc->peer && nc->peer->info->type == NET_CLIENT_DRIVER_VHOST_USER) {
> +            /*
> +             * vhost-user backend is not currently supported for the early
> +             * migration path.
> +             */
> +            n->early_mig = false;
> +        } else {
> +            vdev->migration = g_new0(VirtIODevMigration, 1);
> +            vdev->migration->early_load = false;
> +
> +            vmstate_register_any(VMSTATE_IF(n), &vmstate_virtio_net_early, n);
> +        }
> +    }
>  }
>
>  static void virtio_net_device_unrealize(DeviceState *dev)
> @@ -4090,6 +4136,13 @@ static void virtio_net_device_unrealize(DeviceState *dev)
>      g_free(n->rss_data.indirections_table);
>      net_rx_pkt_uninit(n->rx_pkt);
>      virtio_cleanup(vdev);
> +
> +    if (n->early_mig) {
> +        g_free(vdev->migration);
> +        vdev->migration = NULL;

Nit: You can use g_clear_pointer(vdev->migration, g_free) here.

> +
> +        vmstate_unregister(VMSTATE_IF(n), &vmstate_virtio_net_early, n);
> +    }
>  }
>
>  static void virtio_net_reset(VirtIODevice *vdev)
> diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
> index 8fcf6cfd0b..48de4a430b 100644
> --- a/hw/virtio/virtio.c
> +++ b/hw/virtio/virtio.c
> @@ -3323,6 +3323,7 @@ virtio_load(VirtIODevice *vdev, QEMUFile *f, int version_id)
>      int32_t config_len;
>      uint32_t num;
>      uint32_t features;
> +    bool inconsistent_indices;
>      BusState *qbus = qdev_get_parent_bus(DEVICE(vdev));
>      VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(qbus);
>      VirtioDeviceClass *vdc = VIRTIO_DEVICE_GET_CLASS(vdev);
> @@ -3460,6 +3461,14 @@ virtio_load(VirtIODevice *vdev, QEMUFile *f, int version_id)
>          if (vdev->vq[i].vring.desc) {
>              uint16_t nheads;
>
> +            /*
> +             * Ring indices will be inconsistent for a VMStateDescription
> +             * performing an early load. This shouldn't be an issue as the
> +             * final indices will get sent later once the source has been
> +             * stopped.
> +             */
> +            inconsistent_indices = vdev->migration && vdev->migration->early_load;
> +
>              /*
>               * VIRTIO-1 devices migrate desc, used, and avail ring addresses so
>               * only the region cache needs to be set up.  Legacy devices need
> @@ -3481,12 +3490,15 @@ virtio_load(VirtIODevice *vdev, QEMUFile *f, int version_id)
>
>              nheads = vring_avail_idx(&vdev->vq[i]) - vdev->vq[i].last_avail_idx;
>              /* Check it isn't doing strange things with descriptor numbers. */
> -            if (nheads > vdev->vq[i].vring.num) {
> +            if (!inconsistent_indices && nheads > vdev->vq[i].vring.num) {
>                  virtio_error(vdev, "VQ %d size 0x%x Guest index 0x%x "
>                               "inconsistent with Host index 0x%x: delta 0x%x",
>                               i, vdev->vq[i].vring.num,
>                               vring_avail_idx(&vdev->vq[i]),
>                               vdev->vq[i].last_avail_idx, nheads);
> +                inconsistent_indices = true;
> +            }
> +            if (inconsistent_indices) {
>                  vdev->vq[i].used_idx = 0;
>                  vdev->vq[i].shadow_avail_idx = 0;
>                  vdev->vq[i].inuse = 0;
> diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
> index 6344bd7b68..4c886eb48b 100644
> --- a/include/hw/virtio/virtio.h
> +++ b/include/hw/virtio/virtio.h
> @@ -99,6 +99,14 @@ enum virtio_device_endian {
>      VIRTIO_DEVICE_ENDIAN_BIG,
>  };
>
> +/**
> + * struct VirtIODevMigration - Common VirtIODevice migration structure
> + * @early_load: Flag to indicate an early virtio_load for the device.
> + */
> +typedef struct VirtIODevMigration {
> +    bool early_load;
> +} VirtIODevMigration;
> +
>  /**
>   * struct VirtIODevice - common VirtIO structure
>   * @name: name of the device
> @@ -168,6 +176,7 @@ struct VirtIODevice
>       */
>      EventNotifier config_notifier;
>      bool device_iotlb_enabled;
> +    VirtIODevMigration *migration;

Can we use something like net "struct VirtIONetMigTmp" for this so
VirtIODevice does not need to be expanded?

>  };
>
>  struct VirtioDeviceClass {
> --
> 2.51.0
>

Re: [RFC v2 02/14] virtio,virtio-net: add initial early VMSD for setup-phase migration

[Jonah Palmer]

On 3/24/26 5:27 AM, Eugenio Perez Martin wrote:
> On Fri, Mar 20, 2026 at 3:25 PM Jonah Palmer <jonah.palmer@oracle.com> wrote:
>>
>> Adds a separate VMStateDescription for virtio-net that uses the
>> .early_setup feature. With this feature, we can migrate a virtio-net
>> device's state earlier, before the stop-and-copy phase.
>>
>> Future patches will utilize this to move control plane operations out of
>> the stop-and-copy phase to reduce the downtime latency caused by
>> migrating a virtio-net device.
>>
>> A VirtIODevMigration migration data structure is also introduced here
>> for VirtIODevices to help track the current state of a migration.
>>
>> The early_load member is used to signal that a VirtIODevice is being
>> loaded early and to not throw an error regarding vring indices.
>> Inconsistent indices shouldn't be an issue for a device so long as the
>> final indices are eventually loaded before the device starts.
>>
>> Signed-off-by: Jonah Palmer <jonah.palmer@oracle.com>
>> ---
>>   hw/net/virtio-net.c        | 53 ++++++++++++++++++++++++++++++++++++++
>>   hw/virtio/virtio.c         | 14 +++++++++-
>>   include/hw/virtio/virtio.h |  9 +++++++
>>   3 files changed, 75 insertions(+), 1 deletion(-)
>>
>> diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
>> index 12b3456ca2..ddd6ed6e62 100644
>> --- a/hw/net/virtio-net.c
>> +++ b/hw/net/virtio-net.c
>> @@ -3864,6 +3864,37 @@ static bool failover_hide_primary_device(DeviceListener *listener,
>>       return qatomic_read(&n->failover_primary_hidden);
>>   }
>>
>> +static int virtio_net_early_pre_load(void *opaque)
>> +{
>> +    VirtIONet *n = opaque;
>> +    VirtIODevice *vdev = VIRTIO_DEVICE(n);
>> +
>> +    vdev->migration->early_load = true;
>> +    return 0;
>> +}
>> +
>> +static int virtio_net_early_post_load(void *opaque, int version_id)
>> +{
>> +    VirtIONet *n = opaque;
>> +    VirtIODevice *vdev = VIRTIO_DEVICE(n);
>> +
>> +    vdev->migration->early_load = false;
>> +    return 0;
>> +}
>> +
>> +static const VMStateDescription vmstate_virtio_net_early = {
>> +    .name = "virtio-net-early",
>> +    .minimum_version_id = VIRTIO_NET_VM_VERSION,
>> +    .version_id = VIRTIO_NET_VM_VERSION,
>> +    .early_setup = true,
>> +    .pre_load = virtio_net_early_pre_load,
>> +    .post_load = virtio_net_early_post_load,
>> +    .fields = (const VMStateField[]) {
>> +        VMSTATE_VIRTIO_DEVICE,
>> +        VMSTATE_END_OF_LIST()
>> +    },
>> +};
>> +
>>   static void virtio_net_device_realize(DeviceState *dev, Error **errp)
>>   {
>>       VirtIODevice *vdev = VIRTIO_DEVICE(dev);
>> @@ -4046,6 +4077,21 @@ static void virtio_net_device_realize(DeviceState *dev, Error **errp)
>>               n->rss_data.specified_hash_types.on_bits |
>>               n->rss_data.specified_hash_types.auto_bits;
>>       }
>> +
>> +    if (n->early_mig) {
>> +        if (nc->peer && nc->peer->info->type == NET_CLIENT_DRIVER_VHOST_USER) {
>> +            /*
>> +             * vhost-user backend is not currently supported for the early
>> +             * migration path.
>> +             */
>> +            n->early_mig = false;
>> +        } else {
>> +            vdev->migration = g_new0(VirtIODevMigration, 1);
>> +            vdev->migration->early_load = false;
>> +
>> +            vmstate_register_any(VMSTATE_IF(n), &vmstate_virtio_net_early, n);
>> +        }
>> +    }
>>   }
>>
>>   static void virtio_net_device_unrealize(DeviceState *dev)
>> @@ -4090,6 +4136,13 @@ static void virtio_net_device_unrealize(DeviceState *dev)
>>       g_free(n->rss_data.indirections_table);
>>       net_rx_pkt_uninit(n->rx_pkt);
>>       virtio_cleanup(vdev);
>> +
>> +    if (n->early_mig) {
>> +        g_free(vdev->migration);
>> +        vdev->migration = NULL;
> 
> Nit: You can use g_clear_pointer(vdev->migration, g_free) here.
> 

Ack. Will do!

>> +
>> +        vmstate_unregister(VMSTATE_IF(n), &vmstate_virtio_net_early, n);
>> +    }
>>   }
>>
>>   static void virtio_net_reset(VirtIODevice *vdev)
>> diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
>> index 8fcf6cfd0b..48de4a430b 100644
>> --- a/hw/virtio/virtio.c
>> +++ b/hw/virtio/virtio.c
>> @@ -3323,6 +3323,7 @@ virtio_load(VirtIODevice *vdev, QEMUFile *f, int version_id)
>>       int32_t config_len;
>>       uint32_t num;
>>       uint32_t features;
>> +    bool inconsistent_indices;
>>       BusState *qbus = qdev_get_parent_bus(DEVICE(vdev));
>>       VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(qbus);
>>       VirtioDeviceClass *vdc = VIRTIO_DEVICE_GET_CLASS(vdev);
>> @@ -3460,6 +3461,14 @@ virtio_load(VirtIODevice *vdev, QEMUFile *f, int version_id)
>>           if (vdev->vq[i].vring.desc) {
>>               uint16_t nheads;
>>
>> +            /*
>> +             * Ring indices will be inconsistent for a VMStateDescription
>> +             * performing an early load. This shouldn't be an issue as the
>> +             * final indices will get sent later once the source has been
>> +             * stopped.
>> +             */
>> +            inconsistent_indices = vdev->migration && vdev->migration->early_load;
>> +
>>               /*
>>                * VIRTIO-1 devices migrate desc, used, and avail ring addresses so
>>                * only the region cache needs to be set up.  Legacy devices need
>> @@ -3481,12 +3490,15 @@ virtio_load(VirtIODevice *vdev, QEMUFile *f, int version_id)
>>
>>               nheads = vring_avail_idx(&vdev->vq[i]) - vdev->vq[i].last_avail_idx;
>>               /* Check it isn't doing strange things with descriptor numbers. */
>> -            if (nheads > vdev->vq[i].vring.num) {
>> +            if (!inconsistent_indices && nheads > vdev->vq[i].vring.num) {
>>                   virtio_error(vdev, "VQ %d size 0x%x Guest index 0x%x "
>>                                "inconsistent with Host index 0x%x: delta 0x%x",
>>                                i, vdev->vq[i].vring.num,
>>                                vring_avail_idx(&vdev->vq[i]),
>>                                vdev->vq[i].last_avail_idx, nheads);
>> +                inconsistent_indices = true;
>> +            }
>> +            if (inconsistent_indices) {
>>                   vdev->vq[i].used_idx = 0;
>>                   vdev->vq[i].shadow_avail_idx = 0;
>>                   vdev->vq[i].inuse = 0;
>> diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
>> index 6344bd7b68..4c886eb48b 100644
>> --- a/include/hw/virtio/virtio.h
>> +++ b/include/hw/virtio/virtio.h
>> @@ -99,6 +99,14 @@ enum virtio_device_endian {
>>       VIRTIO_DEVICE_ENDIAN_BIG,
>>   };
>>
>> +/**
>> + * struct VirtIODevMigration - Common VirtIODevice migration structure
>> + * @early_load: Flag to indicate an early virtio_load for the device.
>> + */
>> +typedef struct VirtIODevMigration {
>> +    bool early_load;
>> +} VirtIODevMigration;
>> +
>>   /**
>>    * struct VirtIODevice - common VirtIO structure
>>    * @name: name of the device
>> @@ -168,6 +176,7 @@ struct VirtIODevice
>>        */
>>       EventNotifier config_notifier;
>>       bool device_iotlb_enabled;
>> +    VirtIODevMigration *migration;
> 
> Can we use something like net "struct VirtIONetMigTmp" for this so
> VirtIODevice does not need to be expanded?
> 

I wanted to keep it under VirtIODevice because it's meant to hold 
migration scratch state that's common for all VirtIODevices, not just 
virtio-net. For example, it could be reusable by other virtio devices in 
the future if they were to implement their own early-migration path.

If we made this for virtio-net only, we'd need to modify generic virtio 
code (e.g. an extra callback) just for retrieving the state. I thought 
that this would be the cleaner way to go about handling common 
VirtIODevice state.

>>   };
>>
>>   struct VirtioDeviceClass {
>> --
>> 2.51.0
>>
> 

Re: [RFC v2 02/14] virtio,virtio-net: add initial early VMSD for setup-phase migration

[Eugenio Perez Martin]

On Tue, Mar 24, 2026 at 3:29 PM Jonah Palmer <jonah.palmer@oracle.com> wrote:
>
>
>
> On 3/24/26 5:27 AM, Eugenio Perez Martin wrote:
> > On Fri, Mar 20, 2026 at 3:25 PM Jonah Palmer <jonah.palmer@oracle.com> wrote:
> >>
> >> Adds a separate VMStateDescription for virtio-net that uses the
> >> .early_setup feature. With this feature, we can migrate a virtio-net
> >> device's state earlier, before the stop-and-copy phase.
> >>
> >> Future patches will utilize this to move control plane operations out of
> >> the stop-and-copy phase to reduce the downtime latency caused by
> >> migrating a virtio-net device.
> >>
> >> A VirtIODevMigration migration data structure is also introduced here
> >> for VirtIODevices to help track the current state of a migration.
> >>
> >> The early_load member is used to signal that a VirtIODevice is being
> >> loaded early and to not throw an error regarding vring indices.
> >> Inconsistent indices shouldn't be an issue for a device so long as the
> >> final indices are eventually loaded before the device starts.
> >>
> >> Signed-off-by: Jonah Palmer <jonah.palmer@oracle.com>
> >> ---
> >>   hw/net/virtio-net.c        | 53 ++++++++++++++++++++++++++++++++++++++
> >>   hw/virtio/virtio.c         | 14 +++++++++-
> >>   include/hw/virtio/virtio.h |  9 +++++++
> >>   3 files changed, 75 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
> >> index 12b3456ca2..ddd6ed6e62 100644
> >> --- a/hw/net/virtio-net.c
> >> +++ b/hw/net/virtio-net.c
> >> @@ -3864,6 +3864,37 @@ static bool failover_hide_primary_device(DeviceListener *listener,
> >>       return qatomic_read(&n->failover_primary_hidden);
> >>   }
> >>
> >> +static int virtio_net_early_pre_load(void *opaque)
> >> +{
> >> +    VirtIONet *n = opaque;
> >> +    VirtIODevice *vdev = VIRTIO_DEVICE(n);
> >> +
> >> +    vdev->migration->early_load = true;
> >> +    return 0;
> >> +}
> >> +
> >> +static int virtio_net_early_post_load(void *opaque, int version_id)
> >> +{
> >> +    VirtIONet *n = opaque;
> >> +    VirtIODevice *vdev = VIRTIO_DEVICE(n);
> >> +
> >> +    vdev->migration->early_load = false;
> >> +    return 0;
> >> +}
> >> +
> >> +static const VMStateDescription vmstate_virtio_net_early = {
> >> +    .name = "virtio-net-early",
> >> +    .minimum_version_id = VIRTIO_NET_VM_VERSION,
> >> +    .version_id = VIRTIO_NET_VM_VERSION,
> >> +    .early_setup = true,
> >> +    .pre_load = virtio_net_early_pre_load,
> >> +    .post_load = virtio_net_early_post_load,
> >> +    .fields = (const VMStateField[]) {
> >> +        VMSTATE_VIRTIO_DEVICE,
> >> +        VMSTATE_END_OF_LIST()
> >> +    },
> >> +};
> >> +
> >>   static void virtio_net_device_realize(DeviceState *dev, Error **errp)
> >>   {
> >>       VirtIODevice *vdev = VIRTIO_DEVICE(dev);
> >> @@ -4046,6 +4077,21 @@ static void virtio_net_device_realize(DeviceState *dev, Error **errp)
> >>               n->rss_data.specified_hash_types.on_bits |
> >>               n->rss_data.specified_hash_types.auto_bits;
> >>       }
> >> +
> >> +    if (n->early_mig) {
> >> +        if (nc->peer && nc->peer->info->type == NET_CLIENT_DRIVER_VHOST_USER) {
> >> +            /*
> >> +             * vhost-user backend is not currently supported for the early
> >> +             * migration path.
> >> +             */
> >> +            n->early_mig = false;
> >> +        } else {
> >> +            vdev->migration = g_new0(VirtIODevMigration, 1);
> >> +            vdev->migration->early_load = false;
> >> +
> >> +            vmstate_register_any(VMSTATE_IF(n), &vmstate_virtio_net_early, n);
> >> +        }
> >> +    }
> >>   }
> >>
> >>   static void virtio_net_device_unrealize(DeviceState *dev)
> >> @@ -4090,6 +4136,13 @@ static void virtio_net_device_unrealize(DeviceState *dev)
> >>       g_free(n->rss_data.indirections_table);
> >>       net_rx_pkt_uninit(n->rx_pkt);
> >>       virtio_cleanup(vdev);
> >> +
> >> +    if (n->early_mig) {
> >> +        g_free(vdev->migration);
> >> +        vdev->migration = NULL;
> >
> > Nit: You can use g_clear_pointer(vdev->migration, g_free) here.
> >
>
> Ack. Will do!
>
> >> +
> >> +        vmstate_unregister(VMSTATE_IF(n), &vmstate_virtio_net_early, n);
> >> +    }
> >>   }
> >>
> >>   static void virtio_net_reset(VirtIODevice *vdev)
> >> diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
> >> index 8fcf6cfd0b..48de4a430b 100644
> >> --- a/hw/virtio/virtio.c
> >> +++ b/hw/virtio/virtio.c
> >> @@ -3323,6 +3323,7 @@ virtio_load(VirtIODevice *vdev, QEMUFile *f, int version_id)
> >>       int32_t config_len;
> >>       uint32_t num;
> >>       uint32_t features;
> >> +    bool inconsistent_indices;
> >>       BusState *qbus = qdev_get_parent_bus(DEVICE(vdev));
> >>       VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(qbus);
> >>       VirtioDeviceClass *vdc = VIRTIO_DEVICE_GET_CLASS(vdev);
> >> @@ -3460,6 +3461,14 @@ virtio_load(VirtIODevice *vdev, QEMUFile *f, int version_id)
> >>           if (vdev->vq[i].vring.desc) {
> >>               uint16_t nheads;
> >>
> >> +            /*
> >> +             * Ring indices will be inconsistent for a VMStateDescription
> >> +             * performing an early load. This shouldn't be an issue as the
> >> +             * final indices will get sent later once the source has been
> >> +             * stopped.
> >> +             */
> >> +            inconsistent_indices = vdev->migration && vdev->migration->early_load;
> >> +
> >>               /*
> >>                * VIRTIO-1 devices migrate desc, used, and avail ring addresses so
> >>                * only the region cache needs to be set up.  Legacy devices need
> >> @@ -3481,12 +3490,15 @@ virtio_load(VirtIODevice *vdev, QEMUFile *f, int version_id)
> >>
> >>               nheads = vring_avail_idx(&vdev->vq[i]) - vdev->vq[i].last_avail_idx;
> >>               /* Check it isn't doing strange things with descriptor numbers. */
> >> -            if (nheads > vdev->vq[i].vring.num) {
> >> +            if (!inconsistent_indices && nheads > vdev->vq[i].vring.num) {
> >>                   virtio_error(vdev, "VQ %d size 0x%x Guest index 0x%x "
> >>                                "inconsistent with Host index 0x%x: delta 0x%x",
> >>                                i, vdev->vq[i].vring.num,
> >>                                vring_avail_idx(&vdev->vq[i]),
> >>                                vdev->vq[i].last_avail_idx, nheads);
> >> +                inconsistent_indices = true;
> >> +            }
> >> +            if (inconsistent_indices) {
> >>                   vdev->vq[i].used_idx = 0;
> >>                   vdev->vq[i].shadow_avail_idx = 0;
> >>                   vdev->vq[i].inuse = 0;
> >> diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
> >> index 6344bd7b68..4c886eb48b 100644
> >> --- a/include/hw/virtio/virtio.h
> >> +++ b/include/hw/virtio/virtio.h
> >> @@ -99,6 +99,14 @@ enum virtio_device_endian {
> >>       VIRTIO_DEVICE_ENDIAN_BIG,
> >>   };
> >>
> >> +/**
> >> + * struct VirtIODevMigration - Common VirtIODevice migration structure
> >> + * @early_load: Flag to indicate an early virtio_load for the device.
> >> + */
> >> +typedef struct VirtIODevMigration {
> >> +    bool early_load;
> >> +} VirtIODevMigration;
> >> +
> >>   /**
> >>    * struct VirtIODevice - common VirtIO structure
> >>    * @name: name of the device
> >> @@ -168,6 +176,7 @@ struct VirtIODevice
> >>        */
> >>       EventNotifier config_notifier;
> >>       bool device_iotlb_enabled;
> >> +    VirtIODevMigration *migration;
> >
> > Can we use something like net "struct VirtIONetMigTmp" for this so
> > VirtIODevice does not need to be expanded?
> >
>
> I wanted to keep it under VirtIODevice because it's meant to hold
> migration scratch state that's common for all VirtIODevices, not just
> virtio-net. For example, it could be reusable by other virtio devices in
> the future if they were to implement their own early-migration path.
>
> If we made this for virtio-net only, we'd need to modify generic virtio
> code (e.g. an extra callback) just for retrieving the state. I thought
> that this would be the cleaner way to go about handling common
> VirtIODevice state.
>

I wasn't clear enough. My goal was to move it to a struct allocated
only at migration time, not through all the live of the device.

Moving the allocation and freeing of the struct to
virtio_net_early_pre_load / virtio_net_early_post_load is a first
step. Removing the pointer altogether, the same way VirtIONet doesn't
need a pointer to hold VirtIONetMigTmp, would be even better. But I
see how keeping it live during all the migration may be problematic to
achieve it.

Re: [RFC v2 02/14] virtio,virtio-net: add initial early VMSD for setup-phase migration

[Jonah Palmer]

On 3/24/26 10:38 AM, Eugenio Perez Martin wrote:
> On Tue, Mar 24, 2026 at 3:29 PM Jonah Palmer <jonah.palmer@oracle.com> wrote:
>>
>>
>>
>> On 3/24/26 5:27 AM, Eugenio Perez Martin wrote:
>>> On Fri, Mar 20, 2026 at 3:25 PM Jonah Palmer <jonah.palmer@oracle.com> wrote:
>>>>
>>>> Adds a separate VMStateDescription for virtio-net that uses the
>>>> .early_setup feature. With this feature, we can migrate a virtio-net
>>>> device's state earlier, before the stop-and-copy phase.
>>>>
>>>> Future patches will utilize this to move control plane operations out of
>>>> the stop-and-copy phase to reduce the downtime latency caused by
>>>> migrating a virtio-net device.
>>>>
>>>> A VirtIODevMigration migration data structure is also introduced here
>>>> for VirtIODevices to help track the current state of a migration.
>>>>
>>>> The early_load member is used to signal that a VirtIODevice is being
>>>> loaded early and to not throw an error regarding vring indices.
>>>> Inconsistent indices shouldn't be an issue for a device so long as the
>>>> final indices are eventually loaded before the device starts.
>>>>
>>>> Signed-off-by: Jonah Palmer <jonah.palmer@oracle.com>
>>>> ---
>>>>    hw/net/virtio-net.c        | 53 ++++++++++++++++++++++++++++++++++++++
>>>>    hw/virtio/virtio.c         | 14 +++++++++-
>>>>    include/hw/virtio/virtio.h |  9 +++++++
>>>>    3 files changed, 75 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
>>>> index 12b3456ca2..ddd6ed6e62 100644
>>>> --- a/hw/net/virtio-net.c
>>>> +++ b/hw/net/virtio-net.c
>>>> @@ -3864,6 +3864,37 @@ static bool failover_hide_primary_device(DeviceListener *listener,
>>>>        return qatomic_read(&n->failover_primary_hidden);
>>>>    }
>>>>
>>>> +static int virtio_net_early_pre_load(void *opaque)
>>>> +{
>>>> +    VirtIONet *n = opaque;
>>>> +    VirtIODevice *vdev = VIRTIO_DEVICE(n);
>>>> +
>>>> +    vdev->migration->early_load = true;
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static int virtio_net_early_post_load(void *opaque, int version_id)
>>>> +{
>>>> +    VirtIONet *n = opaque;
>>>> +    VirtIODevice *vdev = VIRTIO_DEVICE(n);
>>>> +
>>>> +    vdev->migration->early_load = false;
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static const VMStateDescription vmstate_virtio_net_early = {
>>>> +    .name = "virtio-net-early",
>>>> +    .minimum_version_id = VIRTIO_NET_VM_VERSION,
>>>> +    .version_id = VIRTIO_NET_VM_VERSION,
>>>> +    .early_setup = true,
>>>> +    .pre_load = virtio_net_early_pre_load,
>>>> +    .post_load = virtio_net_early_post_load,
>>>> +    .fields = (const VMStateField[]) {
>>>> +        VMSTATE_VIRTIO_DEVICE,
>>>> +        VMSTATE_END_OF_LIST()
>>>> +    },
>>>> +};
>>>> +
>>>>    static void virtio_net_device_realize(DeviceState *dev, Error **errp)
>>>>    {
>>>>        VirtIODevice *vdev = VIRTIO_DEVICE(dev);
>>>> @@ -4046,6 +4077,21 @@ static void virtio_net_device_realize(DeviceState *dev, Error **errp)
>>>>                n->rss_data.specified_hash_types.on_bits |
>>>>                n->rss_data.specified_hash_types.auto_bits;
>>>>        }
>>>> +
>>>> +    if (n->early_mig) {
>>>> +        if (nc->peer && nc->peer->info->type == NET_CLIENT_DRIVER_VHOST_USER) {
>>>> +            /*
>>>> +             * vhost-user backend is not currently supported for the early
>>>> +             * migration path.
>>>> +             */
>>>> +            n->early_mig = false;
>>>> +        } else {
>>>> +            vdev->migration = g_new0(VirtIODevMigration, 1);
>>>> +            vdev->migration->early_load = false;
>>>> +
>>>> +            vmstate_register_any(VMSTATE_IF(n), &vmstate_virtio_net_early, n);
>>>> +        }
>>>> +    }
>>>>    }
>>>>
>>>>    static void virtio_net_device_unrealize(DeviceState *dev)
>>>> @@ -4090,6 +4136,13 @@ static void virtio_net_device_unrealize(DeviceState *dev)
>>>>        g_free(n->rss_data.indirections_table);
>>>>        net_rx_pkt_uninit(n->rx_pkt);
>>>>        virtio_cleanup(vdev);
>>>> +
>>>> +    if (n->early_mig) {
>>>> +        g_free(vdev->migration);
>>>> +        vdev->migration = NULL;
>>>
>>> Nit: You can use g_clear_pointer(vdev->migration, g_free) here.
>>>
>>
>> Ack. Will do!
>>
>>>> +
>>>> +        vmstate_unregister(VMSTATE_IF(n), &vmstate_virtio_net_early, n);
>>>> +    }
>>>>    }
>>>>
>>>>    static void virtio_net_reset(VirtIODevice *vdev)
>>>> diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
>>>> index 8fcf6cfd0b..48de4a430b 100644
>>>> --- a/hw/virtio/virtio.c
>>>> +++ b/hw/virtio/virtio.c
>>>> @@ -3323,6 +3323,7 @@ virtio_load(VirtIODevice *vdev, QEMUFile *f, int version_id)
>>>>        int32_t config_len;
>>>>        uint32_t num;
>>>>        uint32_t features;
>>>> +    bool inconsistent_indices;
>>>>        BusState *qbus = qdev_get_parent_bus(DEVICE(vdev));
>>>>        VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(qbus);
>>>>        VirtioDeviceClass *vdc = VIRTIO_DEVICE_GET_CLASS(vdev);
>>>> @@ -3460,6 +3461,14 @@ virtio_load(VirtIODevice *vdev, QEMUFile *f, int version_id)
>>>>            if (vdev->vq[i].vring.desc) {
>>>>                uint16_t nheads;
>>>>
>>>> +            /*
>>>> +             * Ring indices will be inconsistent for a VMStateDescription
>>>> +             * performing an early load. This shouldn't be an issue as the
>>>> +             * final indices will get sent later once the source has been
>>>> +             * stopped.
>>>> +             */
>>>> +            inconsistent_indices = vdev->migration && vdev->migration->early_load;
>>>> +
>>>>                /*
>>>>                 * VIRTIO-1 devices migrate desc, used, and avail ring addresses so
>>>>                 * only the region cache needs to be set up.  Legacy devices need
>>>> @@ -3481,12 +3490,15 @@ virtio_load(VirtIODevice *vdev, QEMUFile *f, int version_id)
>>>>
>>>>                nheads = vring_avail_idx(&vdev->vq[i]) - vdev->vq[i].last_avail_idx;
>>>>                /* Check it isn't doing strange things with descriptor numbers. */
>>>> -            if (nheads > vdev->vq[i].vring.num) {
>>>> +            if (!inconsistent_indices && nheads > vdev->vq[i].vring.num) {
>>>>                    virtio_error(vdev, "VQ %d size 0x%x Guest index 0x%x "
>>>>                                 "inconsistent with Host index 0x%x: delta 0x%x",
>>>>                                 i, vdev->vq[i].vring.num,
>>>>                                 vring_avail_idx(&vdev->vq[i]),
>>>>                                 vdev->vq[i].last_avail_idx, nheads);
>>>> +                inconsistent_indices = true;
>>>> +            }
>>>> +            if (inconsistent_indices) {
>>>>                    vdev->vq[i].used_idx = 0;
>>>>                    vdev->vq[i].shadow_avail_idx = 0;
>>>>                    vdev->vq[i].inuse = 0;
>>>> diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
>>>> index 6344bd7b68..4c886eb48b 100644
>>>> --- a/include/hw/virtio/virtio.h
>>>> +++ b/include/hw/virtio/virtio.h
>>>> @@ -99,6 +99,14 @@ enum virtio_device_endian {
>>>>        VIRTIO_DEVICE_ENDIAN_BIG,
>>>>    };
>>>>
>>>> +/**
>>>> + * struct VirtIODevMigration - Common VirtIODevice migration structure
>>>> + * @early_load: Flag to indicate an early virtio_load for the device.
>>>> + */
>>>> +typedef struct VirtIODevMigration {
>>>> +    bool early_load;
>>>> +} VirtIODevMigration;
>>>> +
>>>>    /**
>>>>     * struct VirtIODevice - common VirtIO structure
>>>>     * @name: name of the device
>>>> @@ -168,6 +176,7 @@ struct VirtIODevice
>>>>         */
>>>>        EventNotifier config_notifier;
>>>>        bool device_iotlb_enabled;
>>>> +    VirtIODevMigration *migration;
>>>
>>> Can we use something like net "struct VirtIONetMigTmp" for this so
>>> VirtIODevice does not need to be expanded?
>>>
>>
>> I wanted to keep it under VirtIODevice because it's meant to hold
>> migration scratch state that's common for all VirtIODevices, not just
>> virtio-net. For example, it could be reusable by other virtio devices in
>> the future if they were to implement their own early-migration path.
>>
>> If we made this for virtio-net only, we'd need to modify generic virtio
>> code (e.g. an extra callback) just for retrieving the state. I thought
>> that this would be the cleaner way to go about handling common
>> VirtIODevice state.
>>
> 
> I wasn't clear enough. My goal was to move it to a struct allocated
> only at migration time, not through all the live of the device.
> 
> Moving the allocation and freeing of the struct to
> virtio_net_early_pre_load / virtio_net_early_post_load is a first
> step. Removing the pointer altogether, the same way VirtIONet doesn't
> need a pointer to hold VirtIONetMigTmp, would be even better. But I
> see how keeping it live during all the migration may be problematic to
> achieve it.
> 

Oh, I see what you're saying now. Right, I agree that, ideally, the 
lifetime of this struct would be during migration only instead of the 
entire device's lifetime. This is something I'll look into.

With regards to keeping VirtIODevMigration out of the VirtIODevice 
struct, this might be a bit trickier. But I think there's some options 
worth exploring that might not require us to do something like this. 
I'll certainly look into this as well!

[RFC v2 03/14] virtio,virtio-net: virtio-delta VMSD - VQ state

[Jonah Palmer]

Introduce a new virtio VMStateDescription vmstate_virtio_delta for
VirtIODevices.

This parent VMSD will be used by virtio devices that send their state
early (before the stop-and-copy phase) and only need to resend any
state that changed since then. Any data that is re-sent is done during
the stop-and-copy phase, when the source has been paused.

This patch also adds subsections to vmstate_virtio_delta for resyncing a
VirtIODevice's VQs' state. Since it's more likely than not that these
fields changed since they were sent early, we just send their updated
values without checking if they did indeed change.

After loading delta VQ state, re-sync runtime shadow queue indices to the
delta-loaded avail indices and clear signalled_used_valid so runtime
queue bookkeeping stays consistent for both split and packed rings.

Lastly, this new VirtIODevice parent VMSD is registered for a virtio-net
device only if it has its early-mig property enabled.

Signed-off-by: Jonah Palmer <jonah.palmer@oracle.com>
---
 hw/net/virtio-net.c        |   2 +
 hw/virtio/virtio.c         | 150 +++++++++++++++++++++++++++++++++++++
 include/hw/virtio/virtio.h |   3 +
 3 files changed, 155 insertions(+)

diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index ddd6ed6e62..5d71ad235e 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -4090,6 +4090,7 @@ static void virtio_net_device_realize(DeviceState *dev, Error **errp)
             vdev->migration->early_load = false;
 
             vmstate_register_any(VMSTATE_IF(n), &vmstate_virtio_net_early, n);
+            virtio_delta_vmsd_register(vdev);
         }
     }
 }
@@ -4142,6 +4143,7 @@ static void virtio_net_device_unrealize(DeviceState *dev)
         vdev->migration = NULL;
 
         vmstate_unregister(VMSTATE_IF(n), &vmstate_virtio_net_early, n);
+        virtio_delta_vmsd_unregister(vdev);
     }
 }
 
diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
index 48de4a430b..47236bf7d7 100644
--- a/hw/virtio/virtio.c
+++ b/hw/virtio/virtio.c
@@ -2780,6 +2780,13 @@ static bool virtio_virtqueue_needed(void *opaque)
     return virtio_host_has_feature(vdev, VIRTIO_F_VERSION_1);
 }
 
+static bool virtio_split_virtqueue_needed(void *opaque)
+{
+    VirtIODevice *vdev = opaque;
+
+    return !virtio_host_has_feature(vdev, VIRTIO_F_RING_PACKED);
+}
+
 static bool virtio_packed_virtqueue_needed(void *opaque)
 {
     VirtIODevice *vdev = opaque;
@@ -2842,6 +2849,18 @@ static const VMStateDescription vmstate_virtqueue = {
     }
 };
 
+static const VMStateDescription vmstate_split_virtqueue = {
+    .name = "split_virtqueue_state",
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .fields = (const VMStateField[]) {
+        VMSTATE_UINT16(last_avail_idx, struct VirtQueue),
+        VMSTATE_UINT16(used_idx, struct VirtQueue),
+        VMSTATE_UINT32(inuse, struct VirtQueue),
+        VMSTATE_END_OF_LIST()
+    }
+};
+
 static const VMStateDescription vmstate_packed_virtqueue = {
     .name = "packed_virtqueue_state",
     .version_id = 1,
@@ -2856,6 +2875,89 @@ static const VMStateDescription vmstate_packed_virtqueue = {
     }
 };
 
+/*
+ * Temporary type used with VMSTATE_WITH_TMP to migrate
+ * active VQs only for VirtIODevices that participated
+ * in early live migration.
+ */
+typedef struct VirtIOMigVQsTmp {
+    VirtIODevice *parent;
+    VirtQueue *vqs;
+    uint16_t num_vqs;
+} VirtIOMigVQsTmp;
+
+static int virtio_delta_vqs_pre_save(void *opaque)
+{
+    VirtIOMigVQsTmp *tmp = opaque;
+    tmp->vqs = tmp->parent->vq;
+    tmp->num_vqs = virtio_get_num_queues(tmp->parent);
+    return 0;
+}
+
+static int virtio_delta_vqs_pre_load(void *opaque)
+{
+    VirtIOMigVQsTmp *tmp = opaque;
+    uint16_t num_vqs = virtio_get_num_queues(tmp->parent);
+
+    tmp->vqs = tmp->parent->vq;
+    tmp->num_vqs = num_vqs;
+
+    if (tmp->num_vqs > VIRTIO_QUEUE_MAX) {
+        return -EINVAL;
+    }
+    return 0;
+}
+
+/* Re-sync runtime shadow queue indices with final delta-loaded indices */
+static int virtio_delta_vqs_post_load(void *opaque, int version_id)
+{
+    VirtIOMigVQsTmp *tmp = opaque;
+    bool packed = virtio_vdev_has_feature(tmp->parent, VIRTIO_F_RING_PACKED);
+    int i;
+
+    for (i = 0; i < tmp->num_vqs; i++) {
+        VirtQueue *vq = &tmp->vqs[i];
+
+        if (!vq->vring.desc) {
+            continue;
+        }
+
+        vq->shadow_avail_idx = vq->last_avail_idx;
+        if (packed) {
+            vq->shadow_avail_wrap_counter = vq->last_avail_wrap_counter;
+        }
+        vq->signalled_used_valid = false;
+    }
+
+    return 0;
+}
+
+static const VMStateDescription vmstate_virtio_delta_split_vqs_tmp = {
+    .name = "virtio-delta/split_vqs_tmp",
+    .pre_save = virtio_delta_vqs_pre_save,
+    .pre_load = virtio_delta_vqs_pre_load,
+    .post_load = virtio_delta_vqs_post_load,
+    .fields = (const VMStateField[]) {
+        VMSTATE_STRUCT_VARRAY_POINTER_UINT16(vqs, VirtIOMigVQsTmp, num_vqs,
+                                             vmstate_split_virtqueue,
+                                             VirtQueue),
+        VMSTATE_END_OF_LIST()
+    },
+};
+
+static const VMStateDescription vmstate_virtio_delta_packed_vqs_tmp = {
+    .name = "virtio-delta/packed_vqs_tmp",
+    .pre_save = virtio_delta_vqs_pre_save,
+    .pre_load = virtio_delta_vqs_pre_load,
+    .post_load = virtio_delta_vqs_post_load,
+    .fields = (const VMStateField[]) {
+        VMSTATE_STRUCT_VARRAY_POINTER_UINT16(vqs, VirtIOMigVQsTmp, num_vqs,
+                                             vmstate_packed_virtqueue,
+                                             VirtQueue),
+        VMSTATE_END_OF_LIST()
+    },
+};
+
 static const VMStateDescription vmstate_virtio_virtqueues = {
     .name = "virtio/virtqueues",
     .version_id = 1,
@@ -3053,6 +3155,54 @@ static const VMStateDescription vmstate_virtio = {
     }
 };
 
+static const VMStateDescription vmstate_virtio_delta_split_virtqueues = {
+    .name = "virtio-delta/split_virtqueues",
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .needed = &virtio_split_virtqueue_needed,
+    .fields = (const VMStateField[]) {
+        VMSTATE_WITH_TMP(VirtIODevice, VirtIOMigVQsTmp,
+                         vmstate_virtio_delta_split_vqs_tmp),
+        VMSTATE_END_OF_LIST()
+    }
+};
+
+static const VMStateDescription vmstate_virtio_delta_packed_virtqueues = {
+    .name = "virtio-delta/packed_virtqueues",
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .needed = &virtio_packed_virtqueue_needed,
+    .fields = (const VMStateField[]) {
+        VMSTATE_WITH_TMP(VirtIODevice, VirtIOMigVQsTmp,
+                         vmstate_virtio_delta_packed_vqs_tmp),
+        VMSTATE_END_OF_LIST()
+    }
+};
+
+static const VMStateDescription vmstate_virtio_delta = {
+    .name = "virtio-delta",
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .fields = (const VMStateField[]) {
+        VMSTATE_END_OF_LIST()
+    },
+    .subsections = (const VMStateDescription * const []) {
+        &vmstate_virtio_delta_split_virtqueues,
+        &vmstate_virtio_delta_packed_virtqueues,
+        NULL
+    }
+};
+
+void virtio_delta_vmsd_register(VirtIODevice *vdev)
+{
+    vmstate_register_any(VMSTATE_IF(vdev), &vmstate_virtio_delta, vdev);
+}
+
+void virtio_delta_vmsd_unregister(VirtIODevice *vdev)
+{
+    vmstate_unregister(VMSTATE_IF(vdev), &vmstate_virtio_delta, vdev);
+}
+
 int virtio_save(VirtIODevice *vdev, QEMUFile *f)
 {
     BusState *qbus = qdev_get_parent_bus(DEVICE(vdev));
diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
index 4c886eb48b..74fed8c324 100644
--- a/include/hw/virtio/virtio.h
+++ b/include/hw/virtio/virtio.h
@@ -305,6 +305,9 @@ int virtqueue_get_avail_bytes(VirtQueue *vq, unsigned int *in_bytes,
 
 void virtio_notify(VirtIODevice *vdev, VirtQueue *vq);
 
+void virtio_delta_vmsd_register(VirtIODevice *vdev);
+void virtio_delta_vmsd_unregister(VirtIODevice *vdev);
+
 int virtio_save(VirtIODevice *vdev, QEMUFile *f);
 
 extern const VMStateInfo virtio_vmstate_info;
-- 
2.51.0

[RFC v2 04/14] virtio-net: detect VirtIODevice status mid-migration change

[Jonah Palmer]

This patch introduces the mechanism in which this series will use to
detect VirtIODevice & VirtIONet state deltas, starting with the
VirtIODevice's 'status' member of a virtio-net device.

Before we send the device's state early, we save each piece of its state
in a temporary structure via virtio_net_early_pre_save. Later, once the
source VM has been paused, we compare the current values of those pieces
to what we saved earlier. If any mismatch is found, virtio-net's VMSD
(vmstate_virtio_net) is enabled and resends all state and device prep
work (as it's normally done today when live migrating a virtio-net
device).

Once all relevant delta checks are in place, a no-delta case will skip
vmstate_virtio_net and only resend updated VQ indices.

For this patch, keep a temporary always-true fallback in
virtio_net_has_delta/virtio_net_needed until follow-up patches cover the
remaining VirtIONet & VirtIODevice deltas.

Signed-off-by: Jonah Palmer <jonah.palmer@oracle.com>
---
 hw/net/virtio-net.c        | 40 ++++++++++++++++++++++++++++++++++++++
 include/hw/virtio/virtio.h |  2 ++
 2 files changed, 42 insertions(+)

diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index 5d71ad235e..2733e0130c 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -3864,6 +3864,16 @@ static bool failover_hide_primary_device(DeviceListener *listener,
     return qatomic_read(&n->failover_primary_hidden);
 }
 
+static int virtio_net_early_pre_save(void *opaque)
+{
+    VirtIONet *n = opaque;
+    VirtIODevice *vdev = VIRTIO_DEVICE(n);
+    VirtIODevMigration *vdev_mig = vdev->migration;
+
+    vdev_mig->status_early = vdev->status;
+    return 0;
+}
+
 static int virtio_net_early_pre_load(void *opaque)
 {
     VirtIONet *n = opaque;
@@ -3887,6 +3897,7 @@ static const VMStateDescription vmstate_virtio_net_early = {
     .minimum_version_id = VIRTIO_NET_VM_VERSION,
     .version_id = VIRTIO_NET_VM_VERSION,
     .early_setup = true,
+    .pre_save = virtio_net_early_pre_save,
     .pre_load = virtio_net_early_pre_load,
     .post_load = virtio_net_early_post_load,
     .fields = (const VMStateField[]) {
@@ -4231,10 +4242,39 @@ static bool dev_unplug_pending(void *opaque)
     return vdc->primary_unplug_pending(dev);
 }
 
+static bool virtio_net_has_delta(VirtIONet *n, VirtIODevice *vdev)
+{
+    VirtIODevMigration *vdev_mig = vdev->migration;
+
+    /* Has the VirtIODevice's status changed? */
+    if (vdev->status != vdev_mig->status_early) {
+        return true;
+    }
+
+    /*
+     * Always return true for now until we're able to detect all possible
+     * changes to a VirtIONet device.
+     */
+    return true;
+}
+
+static bool virtio_net_needed(void *opaque)
+{
+    VirtIONet *n = opaque;
+    VirtIODevice *vdev = VIRTIO_DEVICE(n);
+
+    if (!n->early_mig) {
+        return true;
+    }
+
+    return virtio_net_has_delta(n, vdev);
+}
+
 static const VMStateDescription vmstate_virtio_net = {
     .name = "virtio-net",
     .minimum_version_id = VIRTIO_NET_VM_VERSION,
     .version_id = VIRTIO_NET_VM_VERSION,
+    .needed = virtio_net_needed,
     .fields = (const VMStateField[]) {
         VMSTATE_VIRTIO_DEVICE,
         VMSTATE_END_OF_LIST()
diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
index 74fed8c324..752c46ce53 100644
--- a/include/hw/virtio/virtio.h
+++ b/include/hw/virtio/virtio.h
@@ -102,9 +102,11 @@ enum virtio_device_endian {
 /**
  * struct VirtIODevMigration - Common VirtIODevice migration structure
  * @early_load: Flag to indicate an early virtio_load for the device.
+ * @status_early: Device status at the time it was sent early.
  */
 typedef struct VirtIODevMigration {
     bool early_load;
+    uint8_t status_early;
 } VirtIODevMigration;
 
 /**
-- 
2.51.0

Re: [RFC v2 04/14] virtio-net: detect VirtIODevice status mid-migration change

[Eugenio Perez Martin]

On Fri, Mar 20, 2026 at 3:20 PM Jonah Palmer <jonah.palmer@oracle.com> wrote:
>
> This patch introduces the mechanism in which this series will use to
> detect VirtIODevice & VirtIONet state deltas, starting with the
> VirtIODevice's 'status' member of a virtio-net device.
>
> Before we send the device's state early, we save each piece of its state
> in a temporary structure via virtio_net_early_pre_save. Later, once the
> source VM has been paused,

While I ack this approach should cover 99% of cases—as I think it is
unlikely the guest changes device properties during a live migration—I
still believe these changes should be sent when the guest modifies the
property, before it is paused. That way we shrink the downtime even
more. This comparison fits naturally: Instead of sending one state and
then the deltas at the switchover, we send the state and then the
delta for every change.

However, I'm happy enough with this approach because it is more
self-contained within the migration code. It also reduces the state
sent over the migration channel if the guest changes its device status
many times.

CCing Juraj as we discussed this at the virtio-net upstream meeting.

> we compare the current values of those pieces
> to what we saved earlier. If any mismatch is found, virtio-net's VMSD
> (vmstate_virtio_net) is enabled and resends all state and device prep
> work (as it's normally done today when live migrating a virtio-net
> device).
>
> Once all relevant delta checks are in place, a no-delta case will skip
> vmstate_virtio_net and only resend updated VQ indices.
>
> For this patch, keep a temporary always-true fallback in
> virtio_net_has_delta/virtio_net_needed until follow-up patches cover the
> remaining VirtIONet & VirtIODevice deltas.
>
> Signed-off-by: Jonah Palmer <jonah.palmer@oracle.com>
> ---
>  hw/net/virtio-net.c        | 40 ++++++++++++++++++++++++++++++++++++++
>  include/hw/virtio/virtio.h |  2 ++
>  2 files changed, 42 insertions(+)
>
> diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
> index 5d71ad235e..2733e0130c 100644
> --- a/hw/net/virtio-net.c
> +++ b/hw/net/virtio-net.c
> @@ -3864,6 +3864,16 @@ static bool failover_hide_primary_device(DeviceListener *listener,
>      return qatomic_read(&n->failover_primary_hidden);
>  }
>
> +static int virtio_net_early_pre_save(void *opaque)
> +{
> +    VirtIONet *n = opaque;
> +    VirtIODevice *vdev = VIRTIO_DEVICE(n);
> +    VirtIODevMigration *vdev_mig = vdev->migration;
> +
> +    vdev_mig->status_early = vdev->status;
> +    return 0;
> +}
> +
>  static int virtio_net_early_pre_load(void *opaque)
>  {
>      VirtIONet *n = opaque;
> @@ -3887,6 +3897,7 @@ static const VMStateDescription vmstate_virtio_net_early = {
>      .minimum_version_id = VIRTIO_NET_VM_VERSION,
>      .version_id = VIRTIO_NET_VM_VERSION,
>      .early_setup = true,
> +    .pre_save = virtio_net_early_pre_save,
>      .pre_load = virtio_net_early_pre_load,
>      .post_load = virtio_net_early_post_load,
>      .fields = (const VMStateField[]) {
> @@ -4231,10 +4242,39 @@ static bool dev_unplug_pending(void *opaque)
>      return vdc->primary_unplug_pending(dev);
>  }
>
> +static bool virtio_net_has_delta(VirtIONet *n, VirtIODevice *vdev)
> +{
> +    VirtIODevMigration *vdev_mig = vdev->migration;
> +
> +    /* Has the VirtIODevice's status changed? */
> +    if (vdev->status != vdev_mig->status_early) {
> +        return true;
> +    }

To clarify my point, I think we should add something like this:

if (vdev->status != status) {
  // process it.
  virtio_send_status_update_to_destination()
}

At virtio_net_set_status, or similar.

> +
> +    /*
> +     * Always return true for now until we're able to detect all possible
> +     * changes to a VirtIONet device.
> +     */
> +    return true;
> +}
> +
> +static bool virtio_net_needed(void *opaque)
> +{
> +    VirtIONet *n = opaque;
> +    VirtIODevice *vdev = VIRTIO_DEVICE(n);
> +
> +    if (!n->early_mig) {
> +        return true;
> +    }
> +
> +    return virtio_net_has_delta(n, vdev);
> +}
> +
>  static const VMStateDescription vmstate_virtio_net = {
>      .name = "virtio-net",
>      .minimum_version_id = VIRTIO_NET_VM_VERSION,
>      .version_id = VIRTIO_NET_VM_VERSION,
> +    .needed = virtio_net_needed,
>      .fields = (const VMStateField[]) {
>          VMSTATE_VIRTIO_DEVICE,
>          VMSTATE_END_OF_LIST()
> diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
> index 74fed8c324..752c46ce53 100644
> --- a/include/hw/virtio/virtio.h
> +++ b/include/hw/virtio/virtio.h
> @@ -102,9 +102,11 @@ enum virtio_device_endian {
>  /**
>   * struct VirtIODevMigration - Common VirtIODevice migration structure
>   * @early_load: Flag to indicate an early virtio_load for the device.
> + * @status_early: Device status at the time it was sent early.
>   */
>  typedef struct VirtIODevMigration {
>      bool early_load;
> +    uint8_t status_early;
>  } VirtIODevMigration;
>
>  /**
> --
> 2.51.0
>

Re: [RFC v2 04/14] virtio-net: detect VirtIODevice status mid-migration change

[Jonah Palmer]

On 3/24/26 6:45 AM, Eugenio Perez Martin wrote:
> On Fri, Mar 20, 2026 at 3:20 PM Jonah Palmer <jonah.palmer@oracle.com> wrote:
>>
>> This patch introduces the mechanism in which this series will use to
>> detect VirtIODevice & VirtIONet state deltas, starting with the
>> VirtIODevice's 'status' member of a virtio-net device.
>>
>> Before we send the device's state early, we save each piece of its state
>> in a temporary structure via virtio_net_early_pre_save. Later, once the
>> source VM has been paused,
> 
> While I ack this approach should cover 99% of cases—as I think it is
> unlikely the guest changes device properties during a live migration—I
> still believe these changes should be sent when the guest modifies the
> property, before it is paused. That way we shrink the downtime even
> more. This comparison fits naturally: Instead of sending one state and
> then the deltas at the switchover, we send the state and then the
> delta for every change.
> 
> However, I'm happy enough with this approach because it is more
> self-contained within the migration code. It also reduces the state
> sent over the migration channel if the guest changes its device status
> many times.
> 
> CCing Juraj as we discussed this at the virtio-net upstream meeting.
> 

Right. I agree it would be optimal that, for every change made during 
live migration, the destination is updated with this change and it's not 
left to be handled once the source has been paused, which would impact 
guest-visible downtime (especially if it's an expensive change).

However, the only way to handle mid-migration changes like this is via 
the SaveVMHandlers framework. I did give a stab at this approach in my 
v1 RFC series: 
https://lore.kernel.org/qemu-devel/20250722124127.2497406-1-jonah.palmer@oracle.com/

However, Peter believed that this early-migration work would be better 
suited using the VMStateDescription framework instead. And the VMSD 
framework does not support handling these mid-migration changes before 
the stop-and-copy phase.

>> we compare the current values of those pieces
>> to what we saved earlier. If any mismatch is found, virtio-net's VMSD
>> (vmstate_virtio_net) is enabled and resends all state and device prep
>> work (as it's normally done today when live migrating a virtio-net
>> device).
>>
>> Once all relevant delta checks are in place, a no-delta case will skip
>> vmstate_virtio_net and only resend updated VQ indices.
>>
>> For this patch, keep a temporary always-true fallback in
>> virtio_net_has_delta/virtio_net_needed until follow-up patches cover the
>> remaining VirtIONet & VirtIODevice deltas.
>>
>> Signed-off-by: Jonah Palmer <jonah.palmer@oracle.com>
>> ---
>>   hw/net/virtio-net.c        | 40 ++++++++++++++++++++++++++++++++++++++
>>   include/hw/virtio/virtio.h |  2 ++
>>   2 files changed, 42 insertions(+)
>>
>> diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
>> index 5d71ad235e..2733e0130c 100644
>> --- a/hw/net/virtio-net.c
>> +++ b/hw/net/virtio-net.c
>> @@ -3864,6 +3864,16 @@ static bool failover_hide_primary_device(DeviceListener *listener,
>>       return qatomic_read(&n->failover_primary_hidden);
>>   }
>>
>> +static int virtio_net_early_pre_save(void *opaque)
>> +{
>> +    VirtIONet *n = opaque;
>> +    VirtIODevice *vdev = VIRTIO_DEVICE(n);
>> +    VirtIODevMigration *vdev_mig = vdev->migration;
>> +
>> +    vdev_mig->status_early = vdev->status;
>> +    return 0;
>> +}
>> +
>>   static int virtio_net_early_pre_load(void *opaque)
>>   {
>>       VirtIONet *n = opaque;
>> @@ -3887,6 +3897,7 @@ static const VMStateDescription vmstate_virtio_net_early = {
>>       .minimum_version_id = VIRTIO_NET_VM_VERSION,
>>       .version_id = VIRTIO_NET_VM_VERSION,
>>       .early_setup = true,
>> +    .pre_save = virtio_net_early_pre_save,
>>       .pre_load = virtio_net_early_pre_load,
>>       .post_load = virtio_net_early_post_load,
>>       .fields = (const VMStateField[]) {
>> @@ -4231,10 +4242,39 @@ static bool dev_unplug_pending(void *opaque)
>>       return vdc->primary_unplug_pending(dev);
>>   }
>>
>> +static bool virtio_net_has_delta(VirtIONet *n, VirtIODevice *vdev)
>> +{
>> +    VirtIODevMigration *vdev_mig = vdev->migration;
>> +
>> +    /* Has the VirtIODevice's status changed? */
>> +    if (vdev->status != vdev_mig->status_early) {
>> +        return true;
>> +    }
> 
> To clarify my point, I think we should add something like this:
> 
> if (vdev->status != status) {
>    // process it.
>    virtio_send_status_update_to_destination()
> }
> 
> At virtio_net_set_status, or similar.
> 

See comment above.

>> +
>> +    /*
>> +     * Always return true for now until we're able to detect all possible
>> +     * changes to a VirtIONet device.
>> +     */
>> +    return true;
>> +}
>> +
>> +static bool virtio_net_needed(void *opaque)
>> +{
>> +    VirtIONet *n = opaque;
>> +    VirtIODevice *vdev = VIRTIO_DEVICE(n);
>> +
>> +    if (!n->early_mig) {
>> +        return true;
>> +    }
>> +
>> +    return virtio_net_has_delta(n, vdev);
>> +}
>> +
>>   static const VMStateDescription vmstate_virtio_net = {
>>       .name = "virtio-net",
>>       .minimum_version_id = VIRTIO_NET_VM_VERSION,
>>       .version_id = VIRTIO_NET_VM_VERSION,
>> +    .needed = virtio_net_needed,
>>       .fields = (const VMStateField[]) {
>>           VMSTATE_VIRTIO_DEVICE,
>>           VMSTATE_END_OF_LIST()
>> diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
>> index 74fed8c324..752c46ce53 100644
>> --- a/include/hw/virtio/virtio.h
>> +++ b/include/hw/virtio/virtio.h
>> @@ -102,9 +102,11 @@ enum virtio_device_endian {
>>   /**
>>    * struct VirtIODevMigration - Common VirtIODevice migration structure
>>    * @early_load: Flag to indicate an early virtio_load for the device.
>> + * @status_early: Device status at the time it was sent early.
>>    */
>>   typedef struct VirtIODevMigration {
>>       bool early_load;
>> +    uint8_t status_early;
>>   } VirtIODevMigration;
>>
>>   /**
>> --
>> 2.51.0
>>
> 

Re: [RFC v2 04/14] virtio-net: detect VirtIODevice status mid-migration change

[Eugenio Perez Martin]

On Tue, Mar 24, 2026 at 4:02 PM Jonah Palmer <jonah.palmer@oracle.com> wrote:
>
>
>
> On 3/24/26 6:45 AM, Eugenio Perez Martin wrote:
> > On Fri, Mar 20, 2026 at 3:20 PM Jonah Palmer <jonah.palmer@oracle.com> wrote:
> >>
> >> This patch introduces the mechanism in which this series will use to
> >> detect VirtIODevice & VirtIONet state deltas, starting with the
> >> VirtIODevice's 'status' member of a virtio-net device.
> >>
> >> Before we send the device's state early, we save each piece of its state
> >> in a temporary structure via virtio_net_early_pre_save. Later, once the
> >> source VM has been paused,
> >
> > While I ack this approach should cover 99% of cases—as I think it is
> > unlikely the guest changes device properties during a live migration—I
> > still believe these changes should be sent when the guest modifies the
> > property, before it is paused. That way we shrink the downtime even
> > more. This comparison fits naturally: Instead of sending one state and
> > then the deltas at the switchover, we send the state and then the
> > delta for every change.
> >
> > However, I'm happy enough with this approach because it is more
> > self-contained within the migration code. It also reduces the state
> > sent over the migration channel if the guest changes its device status
> > many times.
> >
> > CCing Juraj as we discussed this at the virtio-net upstream meeting.
> >
>
> Right. I agree it would be optimal that, for every change made during
> live migration, the destination is updated with this change and it's not
> left to be handled once the source has been paused, which would impact
> guest-visible downtime (especially if it's an expensive change).
>
> However, the only way to handle mid-migration changes like this is via
> the SaveVMHandlers framework. I did give a stab at this approach in my
> v1 RFC series:
> https://lore.kernel.org/qemu-devel/20250722124127.2497406-1-jonah.palmer@oracle.com/
>
> However, Peter believed that this early-migration work would be better
> suited using the VMStateDescription framework instead. And the VMSD
> framework does not support handling these mid-migration changes before
> the stop-and-copy phase.
>

Right. But we can extend the VMStateDescription framework to support
asynchronous state sending as well. At first glance, it should just
add a new VMStateDescription to the struct SaveStateEntry, let's call
it live_vmsd, and call it at the same function that calls
SaveVMHandlers->save_live_iterate, which is
migration/savevm.c:qemu_savevm_state_iterate. The load version also
needs to be called when .load_state is called.

While this extends the series, I think little code is needed and I
believe it's the right long-term decision to avoid adding multiple
ways to solve the same problem. Again, is this ".early" version is the
best we can do, I'm pretty sure it solves most of the cases so, sure,
I'm all in :).

[RFC v2 05/14] virtio-net: detect VirtIODevice config buffer mid-migration change

[Jonah Palmer]

This patch saves the config buffer and its length of a virtio-net
device's VirtIODevice to compare with later during the stop-and-copy
phase.

Signed-off-by: Jonah Palmer <jonah.palmer@oracle.com>
---
 hw/net/virtio-net.c        | 21 +++++++++++++++++++++
 include/hw/virtio/virtio.h |  4 ++++
 2 files changed, 25 insertions(+)

diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index 2733e0130c..ca4385df1a 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -3871,6 +3871,15 @@ static int virtio_net_early_pre_save(void *opaque)
     VirtIODevMigration *vdev_mig = vdev->migration;
 
     vdev_mig->status_early = vdev->status;
+
+    /* VirtIODevice config buffer snapshot */
+    g_free(vdev_mig->config_early);
+    vdev_mig->config_len_early = vdev->config_len;
+    if (vdev->config_len) {
+        vdev_mig->config_early = g_memdup2(vdev->config, vdev->config_len);
+    } else {
+        vdev_mig->config_early = NULL;
+    }
     return 0;
 }
 
@@ -4150,6 +4159,9 @@ static void virtio_net_device_unrealize(DeviceState *dev)
     virtio_cleanup(vdev);
 
     if (n->early_mig) {
+        g_free(vdev->migration->config_early);
+        vdev->migration->config_early = NULL;
+
         g_free(vdev->migration);
         vdev->migration = NULL;
 
@@ -4251,6 +4263,15 @@ static bool virtio_net_has_delta(VirtIONet *n, VirtIODevice *vdev)
         return true;
     }
 
+    /* Has the VirtIODevice's config buffer changed? */
+    if (vdev->config_len != vdev_mig->config_len_early) {
+        return true;
+    }
+    if (vdev->config_len && memcmp(vdev->config, vdev_mig->config_early,
+                                   vdev->config_len) != 0) {
+        return true;
+    }
+
     /*
      * Always return true for now until we're able to detect all possible
      * changes to a VirtIONet device.
diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
index 752c46ce53..9949b94b64 100644
--- a/include/hw/virtio/virtio.h
+++ b/include/hw/virtio/virtio.h
@@ -103,10 +103,14 @@ enum virtio_device_endian {
  * struct VirtIODevMigration - Common VirtIODevice migration structure
  * @early_load: Flag to indicate an early virtio_load for the device.
  * @status_early: Device status at the time it was sent early.
+ * @config_len_early: Length of the config buffer at the time it was sent early.
+ * @config_early: Config buffer at the time it was sent early.
  */
 typedef struct VirtIODevMigration {
     bool early_load;
     uint8_t status_early;
+    size_t config_len_early;
+    uint8_t *config_early;
 } VirtIODevMigration;
 
 /**
-- 
2.51.0

Re: [RFC v2 05/14] virtio-net: detect VirtIODevice config buffer mid-migration change

[Eugenio Perez Martin]

On Fri, Mar 20, 2026 at 3:21 PM Jonah Palmer <jonah.palmer@oracle.com> wrote:
>
> This patch saves the config buffer and its length of a virtio-net
> device's VirtIODevice to compare with later during the stop-and-copy
> phase.
>
> Signed-off-by: Jonah Palmer <jonah.palmer@oracle.com>
> ---
>  hw/net/virtio-net.c        | 21 +++++++++++++++++++++
>  include/hw/virtio/virtio.h |  4 ++++
>  2 files changed, 25 insertions(+)
>
> diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
> index 2733e0130c..ca4385df1a 100644
> --- a/hw/net/virtio-net.c
> +++ b/hw/net/virtio-net.c
> @@ -3871,6 +3871,15 @@ static int virtio_net_early_pre_save(void *opaque)
>      VirtIODevMigration *vdev_mig = vdev->migration;
>
>      vdev_mig->status_early = vdev->status;
> +
> +    /* VirtIODevice config buffer snapshot */
> +    g_free(vdev_mig->config_early);
> +    vdev_mig->config_len_early = vdev->config_len;
> +    if (vdev->config_len) {
> +        vdev_mig->config_early = g_memdup2(vdev->config, vdev->config_len);
> +    } else {
> +        vdev_mig->config_early = NULL;
> +    }
>      return 0;
>  }
>
> @@ -4150,6 +4159,9 @@ static void virtio_net_device_unrealize(DeviceState *dev)
>      virtio_cleanup(vdev);
>
>      if (n->early_mig) {
> +        g_free(vdev->migration->config_early);
> +        vdev->migration->config_early = NULL;
> +
>          g_free(vdev->migration);
>          vdev->migration = NULL;
>
> @@ -4251,6 +4263,15 @@ static bool virtio_net_has_delta(VirtIONet *n, VirtIODevice *vdev)
>          return true;
>      }
>
> +    /* Has the VirtIODevice's config buffer changed? */
> +    if (vdev->config_len != vdev_mig->config_len_early) {
> +        return true;
> +    }
> +    if (vdev->config_len && memcmp(vdev->config, vdev_mig->config_early,
> +                                   vdev->config_len) != 0) {

I'm happy with this but maybe a comparison to the config generation is enough?

> +        return true;
> +    }
> +
>      /*
>       * Always return true for now until we're able to detect all possible
>       * changes to a VirtIONet device.
> diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
> index 752c46ce53..9949b94b64 100644
> --- a/include/hw/virtio/virtio.h
> +++ b/include/hw/virtio/virtio.h
> @@ -103,10 +103,14 @@ enum virtio_device_endian {
>   * struct VirtIODevMigration - Common VirtIODevice migration structure
>   * @early_load: Flag to indicate an early virtio_load for the device.
>   * @status_early: Device status at the time it was sent early.
> + * @config_len_early: Length of the config buffer at the time it was sent early.
> + * @config_early: Config buffer at the time it was sent early.
>   */
>  typedef struct VirtIODevMigration {
>      bool early_load;
>      uint8_t status_early;
> +    size_t config_len_early;
> +    uint8_t *config_early;
>  } VirtIODevMigration;
>
>  /**
> --
> 2.51.0
>

Re: [RFC v2 05/14] virtio-net: detect VirtIODevice config buffer mid-migration change

[Jonah Palmer]

On 3/24/26 6:48 AM, Eugenio Perez Martin wrote:
> On Fri, Mar 20, 2026 at 3:21 PM Jonah Palmer <jonah.palmer@oracle.com> wrote:
>>
>> This patch saves the config buffer and its length of a virtio-net
>> device's VirtIODevice to compare with later during the stop-and-copy
>> phase.
>>
>> Signed-off-by: Jonah Palmer <jonah.palmer@oracle.com>
>> ---
>>   hw/net/virtio-net.c        | 21 +++++++++++++++++++++
>>   include/hw/virtio/virtio.h |  4 ++++
>>   2 files changed, 25 insertions(+)
>>
>> diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
>> index 2733e0130c..ca4385df1a 100644
>> --- a/hw/net/virtio-net.c
>> +++ b/hw/net/virtio-net.c
>> @@ -3871,6 +3871,15 @@ static int virtio_net_early_pre_save(void *opaque)
>>       VirtIODevMigration *vdev_mig = vdev->migration;
>>
>>       vdev_mig->status_early = vdev->status;
>> +
>> +    /* VirtIODevice config buffer snapshot */
>> +    g_free(vdev_mig->config_early);
>> +    vdev_mig->config_len_early = vdev->config_len;
>> +    if (vdev->config_len) {
>> +        vdev_mig->config_early = g_memdup2(vdev->config, vdev->config_len);
>> +    } else {
>> +        vdev_mig->config_early = NULL;
>> +    }
>>       return 0;
>>   }
>>
>> @@ -4150,6 +4159,9 @@ static void virtio_net_device_unrealize(DeviceState *dev)
>>       virtio_cleanup(vdev);
>>
>>       if (n->early_mig) {
>> +        g_free(vdev->migration->config_early);
>> +        vdev->migration->config_early = NULL;
>> +
>>           g_free(vdev->migration);
>>           vdev->migration = NULL;
>>
>> @@ -4251,6 +4263,15 @@ static bool virtio_net_has_delta(VirtIONet *n, VirtIODevice *vdev)
>>           return true;
>>       }
>>
>> +    /* Has the VirtIODevice's config buffer changed? */
>> +    if (vdev->config_len != vdev_mig->config_len_early) {
>> +        return true;
>> +    }
>> +    if (vdev->config_len && memcmp(vdev->config, vdev_mig->config_early,
>> +                                   vdev->config_len) != 0) {
> 
> I'm happy with this but maybe a comparison to the config generation is enough?
> 

A change in vdev->generation would only tell us that 
virtio_notify_config ran. It wouldn't tell us which field changed or 
what exactly needs to be updated.

And although this RFC series currently just aims to detect a change and, 
if found, signal for a full reload, an actual follow-up PATCH series for 
this will handle sending the required change to the destination instead 
of doing this full reload. For that, we'll need actual per-field deltas.

>> +        return true;
>> +    }
>> +
>>       /*
>>        * Always return true for now until we're able to detect all possible
>>        * changes to a VirtIONet device.
>> diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
>> index 752c46ce53..9949b94b64 100644
>> --- a/include/hw/virtio/virtio.h
>> +++ b/include/hw/virtio/virtio.h
>> @@ -103,10 +103,14 @@ enum virtio_device_endian {
>>    * struct VirtIODevMigration - Common VirtIODevice migration structure
>>    * @early_load: Flag to indicate an early virtio_load for the device.
>>    * @status_early: Device status at the time it was sent early.
>> + * @config_len_early: Length of the config buffer at the time it was sent early.
>> + * @config_early: Config buffer at the time it was sent early.
>>    */
>>   typedef struct VirtIODevMigration {
>>       bool early_load;
>>       uint8_t status_early;
>> +    size_t config_len_early;
>> +    uint8_t *config_early;
>>   } VirtIODevMigration;
>>
>>   /**
>> --
>> 2.51.0
>>
> 

[RFC v2 06/14] virtio-net: detect VirtIONet MAC addr mid-migration change

[Jonah Palmer]

This patch saves the virtio-net device's MAC address to compare with
later during the stop-and-copy phase.

Also introduce VirtIONetMigration, a per-device migration scratch
structure used to store early snapshots of VirtIONet state for later
delta comparison.

Signed-off-by: Jonah Palmer <jonah.palmer@oracle.com>
---
 hw/net/virtio-net.c            | 15 +++++++++++++++
 include/hw/virtio/virtio-net.h |  9 +++++++++
 2 files changed, 24 insertions(+)

diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index ca4385df1a..4f14bba510 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -3869,6 +3869,7 @@ static int virtio_net_early_pre_save(void *opaque)
     VirtIONet *n = opaque;
     VirtIODevice *vdev = VIRTIO_DEVICE(n);
     VirtIODevMigration *vdev_mig = vdev->migration;
+    VirtIONetMigration *vnet_mig = n->migration;
 
     vdev_mig->status_early = vdev->status;
 
@@ -3880,6 +3881,10 @@ static int virtio_net_early_pre_save(void *opaque)
     } else {
         vdev_mig->config_early = NULL;
     }
+
+    /* VirtIONet MAC info snapshot */
+    memcpy(vnet_mig->mac_early, n->mac, ETH_ALEN);
+
     return 0;
 }
 
@@ -4108,6 +4113,7 @@ static void virtio_net_device_realize(DeviceState *dev, Error **errp)
         } else {
             vdev->migration = g_new0(VirtIODevMigration, 1);
             vdev->migration->early_load = false;
+            n->migration = g_new0(VirtIONetMigration, 1);
 
             vmstate_register_any(VMSTATE_IF(n), &vmstate_virtio_net_early, n);
             virtio_delta_vmsd_register(vdev);
@@ -4165,6 +4171,9 @@ static void virtio_net_device_unrealize(DeviceState *dev)
         g_free(vdev->migration);
         vdev->migration = NULL;
 
+        g_free(n->migration);
+        n->migration = NULL;
+
         vmstate_unregister(VMSTATE_IF(n), &vmstate_virtio_net_early, n);
         virtio_delta_vmsd_unregister(vdev);
     }
@@ -4257,6 +4266,7 @@ static bool dev_unplug_pending(void *opaque)
 static bool virtio_net_has_delta(VirtIONet *n, VirtIODevice *vdev)
 {
     VirtIODevMigration *vdev_mig = vdev->migration;
+    VirtIONetMigration *vnet_mig = n->migration;
 
     /* Has the VirtIODevice's status changed? */
     if (vdev->status != vdev_mig->status_early) {
@@ -4272,6 +4282,11 @@ static bool virtio_net_has_delta(VirtIONet *n, VirtIODevice *vdev)
         return true;
     }
 
+    /* Has the VirtIONet's MAC info changed? */
+    if (memcmp(n->mac, vnet_mig->mac_early, ETH_ALEN) != 0) {
+        return true;
+    }
+
     /*
      * Always return true for now until we're able to detect all possible
      * changes to a VirtIONet device.
diff --git a/include/hw/virtio/virtio-net.h b/include/hw/virtio/virtio-net.h
index ddb141fefc..280155366c 100644
--- a/include/hw/virtio/virtio-net.h
+++ b/include/hw/virtio/virtio-net.h
@@ -167,8 +167,17 @@ typedef struct VirtIONetQueue {
     struct VirtIONet *n;
 } VirtIONetQueue;
 
+/**
+ * struct VirtIONetMigration - VirtIONet migration structure
+ * @mac_early: MAC address early migration snapshot.
+ */
+typedef struct VirtIONetMigration {
+    uint8_t mac_early[ETH_ALEN];
+} VirtIONetMigration;
+
 struct VirtIONet {
     VirtIODevice parent_obj;
+    VirtIONetMigration *migration;
     uint8_t mac[ETH_ALEN];
     uint16_t status;
     VirtIONetQueue *vqs;
-- 
2.51.0

[RFC v2 07/14] virtio-net: detect VirtIONet MAC table mid-migration changes

[Jonah Palmer]

This patch saves the virtio-net device's MAC table information to
compare with later during the stop-and-copy phase.

We leave out mac_table.first_multi since it's derived post-load.

Note: we store the exact mac_table.in_use value and the MIN macro is
only used to bound copy & compare byte spans for safety.

Signed-off-by: Jonah Palmer <jonah.palmer@oracle.com>
---
 hw/net/virtio-net.c            | 34 ++++++++++++++++++++++++++++++++++
 include/hw/virtio/virtio-net.h |  8 ++++++++
 2 files changed, 42 insertions(+)

diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index 4f14bba510..2dd130777e 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -3884,6 +3884,26 @@ static int virtio_net_early_pre_save(void *opaque)
 
     /* VirtIONet MAC info snapshot */
     memcpy(vnet_mig->mac_early, n->mac, ETH_ALEN);
+    vnet_mig->mtable_in_use_early = n->mac_table.in_use;
+    vnet_mig->mtable_uni_overflow_early = n->mac_table.uni_overflow;
+    vnet_mig->mtable_multi_overflow_early = n->mac_table.multi_overflow;
+    /*
+     * Allocate baseline buffer once. Only copy the used slice each time.
+     * This avoids repeated allocations and keeps memcmp fast
+     */
+    if (!vnet_mig->mtable_macs_early) {
+        vnet_mig->mtable_macs_early =
+            g_malloc0(MAC_TABLE_ENTRIES * ETH_ALEN);
+    }
+    /*
+     * Copy used portion only. Cap byte span with MIN for safety even if
+     * a buggy config reports more than capacity.
+     */
+    if (vnet_mig->mtable_in_use_early) {
+        uint32_t used = MIN(vnet_mig->mtable_in_use_early, (uint32_t)MAC_TABLE_ENTRIES);
+        size_t bytes = (size_t)used * ETH_ALEN;
+        memcpy(vnet_mig->mtable_macs_early, n->mac_table.macs, bytes);
+    }
 
     return 0;
 }
@@ -4171,6 +4191,8 @@ static void virtio_net_device_unrealize(DeviceState *dev)
         g_free(vdev->migration);
         vdev->migration = NULL;
 
+        g_free(n->migration->mtable_macs_early);
+        n->migration->mtable_macs_early = NULL;
         g_free(n->migration);
         n->migration = NULL;
 
@@ -4286,6 +4308,18 @@ static bool virtio_net_has_delta(VirtIONet *n, VirtIODevice *vdev)
     if (memcmp(n->mac, vnet_mig->mac_early, ETH_ALEN) != 0) {
         return true;
     }
+    if (n->mac_table.in_use != vnet_mig->mtable_in_use_early ||
+        n->mac_table.uni_overflow != vnet_mig->mtable_uni_overflow_early ||
+        n->mac_table.multi_overflow != vnet_mig->mtable_multi_overflow_early) {
+        return true;
+    }
+    if (n->mac_table.in_use) {
+        uint32_t used = MIN(n->mac_table.in_use, (uint32_t)MAC_TABLE_ENTRIES);
+        size_t bytes = (size_t)used * ETH_ALEN;
+        if (memcmp(n->mac_table.macs, vnet_mig->mtable_macs_early, bytes) != 0) {
+            return true;
+        }
+    }
 
     /*
      * Always return true for now until we're able to detect all possible
diff --git a/include/hw/virtio/virtio-net.h b/include/hw/virtio/virtio-net.h
index 280155366c..1e1f1a3995 100644
--- a/include/hw/virtio/virtio-net.h
+++ b/include/hw/virtio/virtio-net.h
@@ -170,9 +170,17 @@ typedef struct VirtIONetQueue {
 /**
  * struct VirtIONetMigration - VirtIONet migration structure
  * @mac_early: MAC address early migration snapshot.
+ * @mtable_in_use_early: In-use MAC table entries.
+ * @mtable_uni_overflow_early: Unicast overflow MAC table entries.
+ * @mtable_multi_overflow_early: Multicast overflow MAC table entries.
+ * @mtable_macs_early: MAC table entries.
  */
 typedef struct VirtIONetMigration {
     uint8_t mac_early[ETH_ALEN];
+    uint32_t mtable_in_use_early;
+    uint8_t mtable_uni_overflow_early;
+    uint8_t mtable_multi_overflow_early;
+    uint8_t *mtable_macs_early;
 } VirtIONetMigration;
 
 struct VirtIONet {
-- 
2.51.0

[RFC v2 08/14] virtio-net: detect VirtIONet status mid-migration change

[Jonah Palmer]

Save VirtIONet's status field during early migration and compare it
again during stop-and-copy.

VirtIONet keeps its own status bits here, including VIRTIO_NET_S_LINK_UP
and VIRTIO_NET_S_ANNOUNCE.

Signed-off-by: Jonah Palmer <jonah.palmer@oracle.com>
---
 hw/net/virtio-net.c            | 6 ++++++
 include/hw/virtio/virtio-net.h | 2 ++
 2 files changed, 8 insertions(+)

diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index 2dd130777e..88ce33b1d2 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -3872,6 +3872,7 @@ static int virtio_net_early_pre_save(void *opaque)
     VirtIONetMigration *vnet_mig = n->migration;
 
     vdev_mig->status_early = vdev->status;
+    vnet_mig->status_early = n->status;
 
     /* VirtIODevice config buffer snapshot */
     g_free(vdev_mig->config_early);
@@ -4304,6 +4305,11 @@ static bool virtio_net_has_delta(VirtIONet *n, VirtIODevice *vdev)
         return true;
     }
 
+    /* Has the VirtIONet's status changed? */
+    if (n->status != vnet_mig->status_early) {
+        return true;
+    }
+
     /* Has the VirtIONet's MAC info changed? */
     if (memcmp(n->mac, vnet_mig->mac_early, ETH_ALEN) != 0) {
         return true;
diff --git a/include/hw/virtio/virtio-net.h b/include/hw/virtio/virtio-net.h
index 1e1f1a3995..59345f1811 100644
--- a/include/hw/virtio/virtio-net.h
+++ b/include/hw/virtio/virtio-net.h
@@ -169,6 +169,7 @@ typedef struct VirtIONetQueue {
 
 /**
  * struct VirtIONetMigration - VirtIONet migration structure
+ * @status_early: VirtIONet status snapshot.
  * @mac_early: MAC address early migration snapshot.
  * @mtable_in_use_early: In-use MAC table entries.
  * @mtable_uni_overflow_early: Unicast overflow MAC table entries.
@@ -176,6 +177,7 @@ typedef struct VirtIONetQueue {
  * @mtable_macs_early: MAC table entries.
  */
 typedef struct VirtIONetMigration {
+    uint16_t status_early;
     uint8_t mac_early[ETH_ALEN];
     uint32_t mtable_in_use_early;
     uint8_t mtable_uni_overflow_early;
-- 
2.51.0

Re: [RFC v2 08/14] virtio-net: detect VirtIONet status mid-migration change

[Eugenio Perez Martin]

On Fri, Mar 20, 2026 at 3:21 PM Jonah Palmer <jonah.palmer@oracle.com> wrote:
>
> Save VirtIONet's status field during early migration and compare it
> again during stop-and-copy.
>
> VirtIONet keeps its own status bits here, including VIRTIO_NET_S_LINK_UP
> and VIRTIO_NET_S_ANNOUNCE.
>

These are particular to the device, so we don't need to migrate them.
If any, we should check if the destination device changed the link
state so we can update the guest, but we don't do that as far as I
know.

> Signed-off-by: Jonah Palmer <jonah.palmer@oracle.com>
> ---
>  hw/net/virtio-net.c            | 6 ++++++
>  include/hw/virtio/virtio-net.h | 2 ++
>  2 files changed, 8 insertions(+)
>
> diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
> index 2dd130777e..88ce33b1d2 100644
> --- a/hw/net/virtio-net.c
> +++ b/hw/net/virtio-net.c
> @@ -3872,6 +3872,7 @@ static int virtio_net_early_pre_save(void *opaque)
>      VirtIONetMigration *vnet_mig = n->migration;
>
>      vdev_mig->status_early = vdev->status;
> +    vnet_mig->status_early = n->status;
>
>      /* VirtIODevice config buffer snapshot */
>      g_free(vdev_mig->config_early);
> @@ -4304,6 +4305,11 @@ static bool virtio_net_has_delta(VirtIONet *n, VirtIODevice *vdev)
>          return true;
>      }
>
> +    /* Has the VirtIONet's status changed? */
> +    if (n->status != vnet_mig->status_early) {
> +        return true;
> +    }
> +
>      /* Has the VirtIONet's MAC info changed? */
>      if (memcmp(n->mac, vnet_mig->mac_early, ETH_ALEN) != 0) {
>          return true;
> diff --git a/include/hw/virtio/virtio-net.h b/include/hw/virtio/virtio-net.h
> index 1e1f1a3995..59345f1811 100644
> --- a/include/hw/virtio/virtio-net.h
> +++ b/include/hw/virtio/virtio-net.h
> @@ -169,6 +169,7 @@ typedef struct VirtIONetQueue {
>
>  /**
>   * struct VirtIONetMigration - VirtIONet migration structure
> + * @status_early: VirtIONet status snapshot.
>   * @mac_early: MAC address early migration snapshot.
>   * @mtable_in_use_early: In-use MAC table entries.
>   * @mtable_uni_overflow_early: Unicast overflow MAC table entries.
> @@ -176,6 +177,7 @@ typedef struct VirtIONetQueue {
>   * @mtable_macs_early: MAC table entries.
>   */
>  typedef struct VirtIONetMigration {
> +    uint16_t status_early;
>      uint8_t mac_early[ETH_ALEN];
>      uint32_t mtable_in_use_early;
>      uint8_t mtable_uni_overflow_early;
> --
> 2.51.0
>

Re: [RFC v2 08/14] virtio-net: detect VirtIONet status mid-migration change

[Jonah Palmer]

On 3/24/26 7:26 AM, Eugenio Perez Martin wrote:
> On Fri, Mar 20, 2026 at 3:21 PM Jonah Palmer <jonah.palmer@oracle.com> wrote:
>>
>> Save VirtIONet's status field during early migration and compare it
>> again during stop-and-copy.
>>
>> VirtIONet keeps its own status bits here, including VIRTIO_NET_S_LINK_UP
>> and VIRTIO_NET_S_ANNOUNCE.
>>
> 
> These are particular to the device, so we don't need to migrate them.
> If any, we should check if the destination device changed the link
> state so we can update the guest, but we don't do that as far as I
> know.
> 

Sorry I might be misunderstanding something, but isn't VirtIONet's 
status member being migrated today (vmstate_virtio_net_device)?

>> Signed-off-by: Jonah Palmer <jonah.palmer@oracle.com>
>> ---
>>   hw/net/virtio-net.c            | 6 ++++++
>>   include/hw/virtio/virtio-net.h | 2 ++
>>   2 files changed, 8 insertions(+)
>>
>> diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
>> index 2dd130777e..88ce33b1d2 100644
>> --- a/hw/net/virtio-net.c
>> +++ b/hw/net/virtio-net.c
>> @@ -3872,6 +3872,7 @@ static int virtio_net_early_pre_save(void *opaque)
>>       VirtIONetMigration *vnet_mig = n->migration;
>>
>>       vdev_mig->status_early = vdev->status;
>> +    vnet_mig->status_early = n->status;
>>
>>       /* VirtIODevice config buffer snapshot */
>>       g_free(vdev_mig->config_early);
>> @@ -4304,6 +4305,11 @@ static bool virtio_net_has_delta(VirtIONet *n, VirtIODevice *vdev)
>>           return true;
>>       }
>>
>> +    /* Has the VirtIONet's status changed? */
>> +    if (n->status != vnet_mig->status_early) {
>> +        return true;
>> +    }
>> +
>>       /* Has the VirtIONet's MAC info changed? */
>>       if (memcmp(n->mac, vnet_mig->mac_early, ETH_ALEN) != 0) {
>>           return true;
>> diff --git a/include/hw/virtio/virtio-net.h b/include/hw/virtio/virtio-net.h
>> index 1e1f1a3995..59345f1811 100644
>> --- a/include/hw/virtio/virtio-net.h
>> +++ b/include/hw/virtio/virtio-net.h
>> @@ -169,6 +169,7 @@ typedef struct VirtIONetQueue {
>>
>>   /**
>>    * struct VirtIONetMigration - VirtIONet migration structure
>> + * @status_early: VirtIONet status snapshot.
>>    * @mac_early: MAC address early migration snapshot.
>>    * @mtable_in_use_early: In-use MAC table entries.
>>    * @mtable_uni_overflow_early: Unicast overflow MAC table entries.
>> @@ -176,6 +177,7 @@ typedef struct VirtIONetQueue {
>>    * @mtable_macs_early: MAC table entries.
>>    */
>>   typedef struct VirtIONetMigration {
>> +    uint16_t status_early;
>>       uint8_t mac_early[ETH_ALEN];
>>       uint32_t mtable_in_use_early;
>>       uint8_t mtable_uni_overflow_early;
>> --
>> 2.51.0
>>
> 

Re: [RFC v2 08/14] virtio-net: detect VirtIONet status mid-migration change

[Eugenio Perez Martin]

On Tue, Mar 24, 2026 at 5:23 PM Jonah Palmer <jonah.palmer@oracle.com> wrote:
>
>
>
> On 3/24/26 7:26 AM, Eugenio Perez Martin wrote:
> > On Fri, Mar 20, 2026 at 3:21 PM Jonah Palmer <jonah.palmer@oracle.com> wrote:
> >>
> >> Save VirtIONet's status field during early migration and compare it
> >> again during stop-and-copy.
> >>
> >> VirtIONet keeps its own status bits here, including VIRTIO_NET_S_LINK_UP
> >> and VIRTIO_NET_S_ANNOUNCE.
> >>
> >
> > These are particular to the device, so we don't need to migrate them.
> > If any, we should check if the destination device changed the link
> > state so we can update the guest, but we don't do that as far as I
> > know.
> >
>
> Sorry I might be misunderstanding something, but isn't VirtIONet's
> status member being migrated today (vmstate_virtio_net_device)?
>

Nevermind, I was too vhost-vdpa centric with this comment :).
Migrating it in the general case is ok.

> >> Signed-off-by: Jonah Palmer <jonah.palmer@oracle.com>
> >> ---
> >>   hw/net/virtio-net.c            | 6 ++++++
> >>   include/hw/virtio/virtio-net.h | 2 ++
> >>   2 files changed, 8 insertions(+)
> >>
> >> diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
> >> index 2dd130777e..88ce33b1d2 100644
> >> --- a/hw/net/virtio-net.c
> >> +++ b/hw/net/virtio-net.c
> >> @@ -3872,6 +3872,7 @@ static int virtio_net_early_pre_save(void *opaque)
> >>       VirtIONetMigration *vnet_mig = n->migration;
> >>
> >>       vdev_mig->status_early = vdev->status;
> >> +    vnet_mig->status_early = n->status;
> >>
> >>       /* VirtIODevice config buffer snapshot */
> >>       g_free(vdev_mig->config_early);
> >> @@ -4304,6 +4305,11 @@ static bool virtio_net_has_delta(VirtIONet *n, VirtIODevice *vdev)
> >>           return true;
> >>       }
> >>
> >> +    /* Has the VirtIONet's status changed? */
> >> +    if (n->status != vnet_mig->status_early) {
> >> +        return true;
> >> +    }
> >> +
> >>       /* Has the VirtIONet's MAC info changed? */
> >>       if (memcmp(n->mac, vnet_mig->mac_early, ETH_ALEN) != 0) {
> >>           return true;
> >> diff --git a/include/hw/virtio/virtio-net.h b/include/hw/virtio/virtio-net.h
> >> index 1e1f1a3995..59345f1811 100644
> >> --- a/include/hw/virtio/virtio-net.h
> >> +++ b/include/hw/virtio/virtio-net.h
> >> @@ -169,6 +169,7 @@ typedef struct VirtIONetQueue {
> >>
> >>   /**
> >>    * struct VirtIONetMigration - VirtIONet migration structure
> >> + * @status_early: VirtIONet status snapshot.
> >>    * @mac_early: MAC address early migration snapshot.
> >>    * @mtable_in_use_early: In-use MAC table entries.
> >>    * @mtable_uni_overflow_early: Unicast overflow MAC table entries.
> >> @@ -176,6 +177,7 @@ typedef struct VirtIONetQueue {
> >>    * @mtable_macs_early: MAC table entries.
> >>    */
> >>   typedef struct VirtIONetMigration {
> >> +    uint16_t status_early;
> >>       uint8_t mac_early[ETH_ALEN];
> >>       uint32_t mtable_in_use_early;
> >>       uint8_t mtable_uni_overflow_early;
> >> --
> >> 2.51.0
> >>
> >
>

[RFC v2 09/14] virtio-net: detect VirtIONet Rx filter mid-migration changes

[Jonah Palmer]

This patch saves the virtio-net device's Rx filter flags as a bit-packed
uint8_t to compare with later during the stop-and-copy phase.

A local enum VirtIONetRxFlags and static inline virtio_net_rx_flags_pack
function is introduced to bit-pack VirtIONet's promisc, allmulti,
alluni, nomulti, nouni, and nobcast into a single byte for easier
comparison.

Signed-off-by: Jonah Palmer <jonah.palmer@oracle.com>
---
 hw/net/virtio-net.c            | 28 ++++++++++++++++++++++++++++
 include/hw/virtio/virtio-net.h |  2 ++
 2 files changed, 30 insertions(+)

diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index 88ce33b1d2..42c585142d 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -3864,6 +3864,26 @@ static bool failover_hide_primary_device(DeviceListener *listener,
     return qatomic_read(&n->failover_primary_hidden);
 }
 
+enum VirtIONetRxFlags {
+    VNET_RX_F_PROMISC  = 1u << 0,
+    VNET_RX_F_ALLMULTI = 1u << 1,
+    VNET_RX_F_ALLUNI   = 1u << 2,
+    VNET_RX_F_NOMULTI  = 1u << 3,
+    VNET_RX_F_NOUNI    = 1u << 4,
+    VNET_RX_F_NOBCAST  = 1u << 5,
+};
+
+/* Pack current Rx filter flags into a single uint8_t for comparison */
+static inline uint8_t virtio_net_rx_flags_pack(const VirtIONet *n)
+{
+    return (n->promisc  ? VNET_RX_F_PROMISC  : 0) |
+           (n->allmulti ? VNET_RX_F_ALLMULTI : 0) |
+           (n->alluni   ? VNET_RX_F_ALLUNI   : 0) |
+           (n->nomulti  ? VNET_RX_F_NOMULTI  : 0) |
+           (n->nouni    ? VNET_RX_F_NOUNI    : 0) |
+           (n->nobcast  ? VNET_RX_F_NOBCAST  : 0);
+}
+
 static int virtio_net_early_pre_save(void *opaque)
 {
     VirtIONet *n = opaque;
@@ -3906,6 +3926,9 @@ static int virtio_net_early_pre_save(void *opaque)
         memcpy(vnet_mig->mtable_macs_early, n->mac_table.macs, bytes);
     }
 
+    /* Rx filter flags snapshot */
+    vnet_mig->rx_flags_early = virtio_net_rx_flags_pack(n);
+
     return 0;
 }
 
@@ -4327,6 +4350,11 @@ static bool virtio_net_has_delta(VirtIONet *n, VirtIODevice *vdev)
         }
     }
 
+    /* Has the VirtIONet's Rx filter flags changed? */
+    if (virtio_net_rx_flags_pack(n) != vnet_mig->rx_flags_early) {
+        return true;
+    }
+
     /*
      * Always return true for now until we're able to detect all possible
      * changes to a VirtIONet device.
diff --git a/include/hw/virtio/virtio-net.h b/include/hw/virtio/virtio-net.h
index 59345f1811..9135d277ff 100644
--- a/include/hw/virtio/virtio-net.h
+++ b/include/hw/virtio/virtio-net.h
@@ -175,6 +175,7 @@ typedef struct VirtIONetQueue {
  * @mtable_uni_overflow_early: Unicast overflow MAC table entries.
  * @mtable_multi_overflow_early: Multicast overflow MAC table entries.
  * @mtable_macs_early: MAC table entries.
+ * @rx_flags_early: Bit-packed RX filters (promisc, allmulti, alluni, etc.).
  */
 typedef struct VirtIONetMigration {
     uint16_t status_early;
@@ -183,6 +184,7 @@ typedef struct VirtIONetMigration {
     uint8_t mtable_uni_overflow_early;
     uint8_t mtable_multi_overflow_early;
     uint8_t *mtable_macs_early;
+    uint8_t rx_flags_early;
 } VirtIONetMigration;
 
 struct VirtIONet {
-- 
2.51.0

[RFC v2 10/14] virtio-net: detect VirtIONet VLAN filter table changes

[Jonah Palmer]

Save the current state of virtio-net's VLAN filter table at early
migration time to compare with later during the stop-and-copy phase.

Signed-off-by: Jonah Palmer <jonah.palmer@oracle.com>
---
 hw/net/virtio-net.c            | 14 ++++++++++++++
 include/hw/virtio/virtio-net.h |  2 ++
 2 files changed, 16 insertions(+)

diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index 42c585142d..b3f71d8f84 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -3890,6 +3890,7 @@ static int virtio_net_early_pre_save(void *opaque)
     VirtIODevice *vdev = VIRTIO_DEVICE(n);
     VirtIODevMigration *vdev_mig = vdev->migration;
     VirtIONetMigration *vnet_mig = n->migration;
+    size_t vlans_size = (size_t)(MAX_VLAN >> 3);
 
     vdev_mig->status_early = vdev->status;
     vnet_mig->status_early = n->status;
@@ -3929,6 +3930,12 @@ static int virtio_net_early_pre_save(void *opaque)
     /* Rx filter flags snapshot */
     vnet_mig->rx_flags_early = virtio_net_rx_flags_pack(n);
 
+    /* VLAN filter table snapshot */
+    if (!vnet_mig->vlans_early) {
+        vnet_mig->vlans_early = g_malloc0(vlans_size);
+    }
+    memcpy(vnet_mig->vlans_early, n->vlans, vlans_size);
+
     return 0;
 }
 
@@ -4217,6 +4224,8 @@ static void virtio_net_device_unrealize(DeviceState *dev)
 
         g_free(n->migration->mtable_macs_early);
         n->migration->mtable_macs_early = NULL;
+        g_free(n->migration->vlans_early);
+        n->migration->vlans_early = NULL;
         g_free(n->migration);
         n->migration = NULL;
 
@@ -4355,6 +4364,11 @@ static bool virtio_net_has_delta(VirtIONet *n, VirtIODevice *vdev)
         return true;
     }
 
+    /* Has the VirtIONet's VLAN filter table changed? */
+    if (memcmp(n->vlans, vnet_mig->vlans_early, MAX_VLAN >> 3) != 0) {
+        return true;
+    }
+
     /*
      * Always return true for now until we're able to detect all possible
      * changes to a VirtIONet device.
diff --git a/include/hw/virtio/virtio-net.h b/include/hw/virtio/virtio-net.h
index 9135d277ff..5df2dd0513 100644
--- a/include/hw/virtio/virtio-net.h
+++ b/include/hw/virtio/virtio-net.h
@@ -176,6 +176,7 @@ typedef struct VirtIONetQueue {
  * @mtable_multi_overflow_early: Multicast overflow MAC table entries.
  * @mtable_macs_early: MAC table entries.
  * @rx_flags_early: Bit-packed RX filters (promisc, allmulti, alluni, etc.).
+ * @vlans_early: VLAN filter table snapshot.
  */
 typedef struct VirtIONetMigration {
     uint16_t status_early;
@@ -185,6 +186,7 @@ typedef struct VirtIONetMigration {
     uint8_t mtable_multi_overflow_early;
     uint8_t *mtable_macs_early;
     uint8_t rx_flags_early;
+    uint8_t *vlans_early;
 } VirtIONetMigration;
 
 struct VirtIONet {
-- 
2.51.0

[RFC v2 11/14] virtio-net: detect VirtIONet guest offload & MQ mid-migration changes

[Jonah Palmer]

Save the state of virtio-net's guest offloads and multiqueue
configuration at early migration time to compare with later during
the stop-and-copy phase.

Signed-off-by: Jonah Palmer <jonah.palmer@oracle.com>
---
 hw/net/virtio-net.c            | 20 ++++++++++++++++++++
 include/hw/virtio/virtio-net.h |  6 ++++++
 2 files changed, 26 insertions(+)

diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index b3f71d8f84..2c0b42debb 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -3936,6 +3936,13 @@ static int virtio_net_early_pre_save(void *opaque)
     }
     memcpy(vnet_mig->vlans_early, n->vlans, vlans_size);
 
+    /* Guest offloads snapshot */
+    vnet_mig->guest_offloads_early = n->curr_guest_offloads;
+
+    /* Multiqueue state snapshot */
+    vnet_mig->mq_early = n->multiqueue;
+    vnet_mig->queue_pairs_early = n->curr_queue_pairs;
+
     return 0;
 }
 
@@ -4369,6 +4376,19 @@ static bool virtio_net_has_delta(VirtIONet *n, VirtIODevice *vdev)
         return true;
     }
 
+    /* Has the VirtIONet's guest offloads changed? */
+    if (n->curr_guest_offloads != vnet_mig->guest_offloads_early) {
+        return true;
+    }
+
+    /* Has the VirtIONet's multiqueue state changed? */
+    if (n->multiqueue != vnet_mig->mq_early) {
+        return true;
+    }
+    if (n->curr_queue_pairs != vnet_mig->queue_pairs_early) {
+        return true;
+    }
+
     /*
      * Always return true for now until we're able to detect all possible
      * changes to a VirtIONet device.
diff --git a/include/hw/virtio/virtio-net.h b/include/hw/virtio/virtio-net.h
index 5df2dd0513..5d6179fa19 100644
--- a/include/hw/virtio/virtio-net.h
+++ b/include/hw/virtio/virtio-net.h
@@ -177,6 +177,9 @@ typedef struct VirtIONetQueue {
  * @mtable_macs_early: MAC table entries.
  * @rx_flags_early: Bit-packed RX filters (promisc, allmulti, alluni, etc.).
  * @vlans_early: VLAN filter table snapshot.
+ * @guest_offloads_early: Guest offloads snapshot.
+ * @mq_early: Multiqueue state snapshot.
+ * @queue_pairs_early: Queue pairs snapshot.
  */
 typedef struct VirtIONetMigration {
     uint16_t status_early;
@@ -187,6 +190,9 @@ typedef struct VirtIONetMigration {
     uint8_t *mtable_macs_early;
     uint8_t rx_flags_early;
     uint8_t *vlans_early;
+    uint64_t guest_offloads_early;
+    int mq_early;
+    uint16_t queue_pairs_early;
 } VirtIONetMigration;
 
 struct VirtIONet {
-- 
2.51.0

[RFC v2 12/14] virtio-net: detect RSS state mid-migration changes

[Jonah Palmer]

Save the current RSS state at early migration time to compare with
later during the stop-and-copy phase.

Only the RSS state that can change mid-migration is tracked.

RSS fields derived from backend capability/feature negotiation (for
example software-RSS capability flags and supported/peer hash capability
metadata) are intentionally excluded from delta checks.

Signed-off-by: Jonah Palmer <jonah.palmer@oracle.com>
---
 hw/net/virtio-net.c            | 50 ++++++++++++++++++++++++++++++++++
 include/hw/virtio/virtio-net.h | 16 +++++++++++
 2 files changed, 66 insertions(+)

diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index 2c0b42debb..3ee49a043a 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -3943,6 +3943,27 @@ static int virtio_net_early_pre_save(void *opaque)
     vnet_mig->mq_early = n->multiqueue;
     vnet_mig->queue_pairs_early = n->curr_queue_pairs;
 
+    /* RSS state snapshot */
+    vnet_mig->rss_enabled_early = n->rss_data.enabled;
+    vnet_mig->rss_redirect_early = n->rss_data.redirect;
+    vnet_mig->rss_populate_hash_early = n->rss_data.populate_hash;
+    vnet_mig->rss_runtime_hash_types_early = n->rss_data.runtime_hash_types;
+    vnet_mig->rss_indirections_len_early = n->rss_data.indirections_len;
+    vnet_mig->rss_default_queue_early = n->rss_data.default_queue;
+    memcpy(vnet_mig->rss_key_early, n->rss_data.key, VIRTIO_NET_RSS_MAX_KEY_SIZE);
+
+    /* Snapshot RSS indirections table if present */
+    g_free(vnet_mig->rss_indirections_table_early);
+    vnet_mig->rss_indirections_table_early = NULL;
+
+    if (n->rss_data.indirections_len && n->rss_data.indirections_table) {
+        /* Casting to size_t avoids implicit narrow/widen arithmetic */
+        size_t bytes = (size_t)n->rss_data.indirections_len * sizeof(uint16_t);
+
+        vnet_mig->rss_indirections_table_early =
+            g_memdup2(n->rss_data.indirections_table, bytes);
+    }
+
     return 0;
 }
 
@@ -4233,6 +4254,8 @@ static void virtio_net_device_unrealize(DeviceState *dev)
         n->migration->mtable_macs_early = NULL;
         g_free(n->migration->vlans_early);
         n->migration->vlans_early = NULL;
+        g_free(n->migration->rss_indirections_table_early);
+        n->migration->rss_indirections_table_early = NULL;
         g_free(n->migration);
         n->migration = NULL;
 
@@ -4389,6 +4412,33 @@ static bool virtio_net_has_delta(VirtIONet *n, VirtIODevice *vdev)
         return true;
     }
 
+    /* Has the VirtIONet's RSS state changed? */
+    if (n->rss_data.enabled != vnet_mig->rss_enabled_early ||
+        n->rss_data.redirect != vnet_mig->rss_redirect_early ||
+        n->rss_data.populate_hash != vnet_mig->rss_populate_hash_early ||
+        n->rss_data.runtime_hash_types != vnet_mig->rss_runtime_hash_types_early ||
+        n->rss_data.indirections_len != vnet_mig->rss_indirections_len_early ||
+        n->rss_data.default_queue != vnet_mig->rss_default_queue_early) {
+        return true;
+    }
+    if (memcmp(n->rss_data.key, vnet_mig->rss_key_early,
+               VIRTIO_NET_RSS_MAX_KEY_SIZE) != 0) {
+        return true;
+    }
+    if (n->rss_data.indirections_len) {
+        size_t bytes = (size_t)n->rss_data.indirections_len * sizeof(uint16_t);
+
+        /* If either side lacks a buffer when len > 0, treat as changed */
+        if (!n->rss_data.indirections_table ||
+            !vnet_mig->rss_indirections_table_early) {
+            return true;
+        }
+        if (memcmp(n->rss_data.indirections_table,
+                   vnet_mig->rss_indirections_table_early, bytes) != 0) {
+            return true;
+        }
+    }
+
     /*
      * Always return true for now until we're able to detect all possible
      * changes to a VirtIONet device.
diff --git a/include/hw/virtio/virtio-net.h b/include/hw/virtio/virtio-net.h
index 5d6179fa19..88074a0976 100644
--- a/include/hw/virtio/virtio-net.h
+++ b/include/hw/virtio/virtio-net.h
@@ -180,6 +180,14 @@ typedef struct VirtIONetQueue {
  * @guest_offloads_early: Guest offloads snapshot.
  * @mq_early: Multiqueue state snapshot.
  * @queue_pairs_early: Queue pairs snapshot.
+ * @rss_enabled_early: RSS enabled flag.
+ * @rss_redirect_early: RSS redirect flag.
+ * @rss_populate_hash_early: RSS populate hash flag.
+ * @rss_runtime_hash_types_early: RSS runtime hash types.
+ * @rss_indirections_len_early: RSS indirections length.
+ * @rss_default_queue_early: RSS default queue.
+ * @rss_key_early: RSS key.
+ * @rss_indirections_table_early: RSS indirections table.
  */
 typedef struct VirtIONetMigration {
     uint16_t status_early;
@@ -193,6 +201,14 @@ typedef struct VirtIONetMigration {
     uint64_t guest_offloads_early;
     int mq_early;
     uint16_t queue_pairs_early;
+    bool rss_enabled_early;
+    bool rss_redirect_early;
+    bool rss_populate_hash_early;
+    uint32_t rss_runtime_hash_types_early;
+    uint16_t rss_indirections_len_early;
+    uint16_t rss_default_queue_early;
+    uint8_t rss_key_early[VIRTIO_NET_RSS_MAX_KEY_SIZE];
+    uint16_t *rss_indirections_table_early;
 } VirtIONetMigration;
 
 struct VirtIONet {
-- 
2.51.0

[RFC v2 13/14] virtio-net: detect pending Tx work for VQs mid-migration changes

[Jonah Palmer]

Track per-queue pending Tx work state during early migration time.

This includes a snapshot of pending Tx state for active queue pairs,
checking for deltas at the end of migration, and freeing the snapshot
buffer during device unrealize.

With this final delta signal in place, drop the temporary always-true
fallback in virtio_net_has_delta and return false when no deltas are
found.

Signed-off-by: Jonah Palmer <jonah.palmer@oracle.com>
---
 hw/net/virtio-net.c            | 26 +++++++++++++++++++++-----
 include/hw/virtio/virtio-net.h |  2 ++
 2 files changed, 23 insertions(+), 5 deletions(-)

diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index 3ee49a043a..483a43be4f 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -3943,6 +3943,14 @@ static int virtio_net_early_pre_save(void *opaque)
     vnet_mig->mq_early = n->multiqueue;
     vnet_mig->queue_pairs_early = n->curr_queue_pairs;
 
+    /* Tx waiting snapshot for active queue pairs */
+    if (!vnet_mig->tx_waiting_early) {
+        vnet_mig->tx_waiting_early = g_new0(uint32_t, n->max_queue_pairs);
+    }
+    for (int i = 0; i < n->curr_queue_pairs; i++) {
+        vnet_mig->tx_waiting_early[i] = n->vqs[i].tx_waiting;
+    }
+
     /* RSS state snapshot */
     vnet_mig->rss_enabled_early = n->rss_data.enabled;
     vnet_mig->rss_redirect_early = n->rss_data.redirect;
@@ -4254,6 +4262,8 @@ static void virtio_net_device_unrealize(DeviceState *dev)
         n->migration->mtable_macs_early = NULL;
         g_free(n->migration->vlans_early);
         n->migration->vlans_early = NULL;
+        g_free(n->migration->tx_waiting_early);
+        n->migration->tx_waiting_early = NULL;
         g_free(n->migration->rss_indirections_table_early);
         n->migration->rss_indirections_table_early = NULL;
         g_free(n->migration);
@@ -4412,6 +4422,16 @@ static bool virtio_net_has_delta(VirtIONet *n, VirtIODevice *vdev)
         return true;
     }
 
+    /* Has any active queue's tx_waiting changed? */
+    if (!vnet_mig->tx_waiting_early) {
+        return true;
+    }
+    for (int i = 0; i < n->curr_queue_pairs; i++) {
+        if (n->vqs[i].tx_waiting != vnet_mig->tx_waiting_early[i]) {
+            return true;
+        }
+    }
+
     /* Has the VirtIONet's RSS state changed? */
     if (n->rss_data.enabled != vnet_mig->rss_enabled_early ||
         n->rss_data.redirect != vnet_mig->rss_redirect_early ||
@@ -4439,11 +4459,7 @@ static bool virtio_net_has_delta(VirtIONet *n, VirtIODevice *vdev)
         }
     }
 
-    /*
-     * Always return true for now until we're able to detect all possible
-     * changes to a VirtIONet device.
-     */
-    return true;
+    return false;
 }
 
 static bool virtio_net_needed(void *opaque)
diff --git a/include/hw/virtio/virtio-net.h b/include/hw/virtio/virtio-net.h
index 88074a0976..dbbacc83bb 100644
--- a/include/hw/virtio/virtio-net.h
+++ b/include/hw/virtio/virtio-net.h
@@ -180,6 +180,7 @@ typedef struct VirtIONetQueue {
  * @guest_offloads_early: Guest offloads snapshot.
  * @mq_early: Multiqueue state snapshot.
  * @queue_pairs_early: Queue pairs snapshot.
+ * @tx_waiting_early: Per-queue pending-Tx snapshot.
  * @rss_enabled_early: RSS enabled flag.
  * @rss_redirect_early: RSS redirect flag.
  * @rss_populate_hash_early: RSS populate hash flag.
@@ -201,6 +202,7 @@ typedef struct VirtIONetMigration {
     uint64_t guest_offloads_early;
     int mq_early;
     uint16_t queue_pairs_early;
+    uint32_t *tx_waiting_early;
     bool rss_enabled_early;
     bool rss_redirect_early;
     bool rss_populate_hash_early;
-- 
2.51.0

Re: [RFC v2 13/14] virtio-net: detect pending Tx work for VQs mid-migration changes

[Eugenio Perez Martin]

On Fri, Mar 20, 2026 at 3:21 PM Jonah Palmer <jonah.palmer@oracle.com> wrote:
>
> Track per-queue pending Tx work state during early migration time.
>
> This includes a snapshot of pending Tx state for active queue pairs,
> checking for deltas at the end of migration, and freeing the snapshot
> buffer during device unrealize.
>
> With this final delta signal in place, drop the temporary always-true
> fallback in virtio_net_has_delta and return false when no deltas are
> found.
>
> Signed-off-by: Jonah Palmer <jonah.palmer@oracle.com>
> ---
>  hw/net/virtio-net.c            | 26 +++++++++++++++++++++-----
>  include/hw/virtio/virtio-net.h |  2 ++
>  2 files changed, 23 insertions(+), 5 deletions(-)
>
> diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
> index 3ee49a043a..483a43be4f 100644
> --- a/hw/net/virtio-net.c
> +++ b/hw/net/virtio-net.c
> @@ -3943,6 +3943,14 @@ static int virtio_net_early_pre_save(void *opaque)
>      vnet_mig->mq_early = n->multiqueue;
>      vnet_mig->queue_pairs_early = n->curr_queue_pairs;
>
> +    /* Tx waiting snapshot for active queue pairs */
> +    if (!vnet_mig->tx_waiting_early) {
> +        vnet_mig->tx_waiting_early = g_new0(uint32_t, n->max_queue_pairs);
> +    }
> +    for (int i = 0; i < n->curr_queue_pairs; i++) {
> +        vnet_mig->tx_waiting_early[i] = n->vqs[i].tx_waiting;
> +    }
> +
>      /* RSS state snapshot */
>      vnet_mig->rss_enabled_early = n->rss_data.enabled;
>      vnet_mig->rss_redirect_early = n->rss_data.redirect;
> @@ -4254,6 +4262,8 @@ static void virtio_net_device_unrealize(DeviceState *dev)
>          n->migration->mtable_macs_early = NULL;
>          g_free(n->migration->vlans_early);
>          n->migration->vlans_early = NULL;
> +        g_free(n->migration->tx_waiting_early);
> +        n->migration->tx_waiting_early = NULL;
>          g_free(n->migration->rss_indirections_table_early);
>          n->migration->rss_indirections_table_early = NULL;
>          g_free(n->migration);
> @@ -4412,6 +4422,16 @@ static bool virtio_net_has_delta(VirtIONet *n, VirtIODevice *vdev)
>          return true;
>      }
>
> +    /* Has any active queue's tx_waiting changed? */
> +    if (!vnet_mig->tx_waiting_early) {
> +        return true;
> +    }
> +    for (int i = 0; i < n->curr_queue_pairs; i++) {
> +        if (n->vqs[i].tx_waiting != vnet_mig->tx_waiting_early[i]) {
> +            return true;
> +        }
> +    }
> +
>      /* Has the VirtIONet's RSS state changed? */
>      if (n->rss_data.enabled != vnet_mig->rss_enabled_early ||
>          n->rss_data.redirect != vnet_mig->rss_redirect_early ||
> @@ -4439,11 +4459,7 @@ static bool virtio_net_has_delta(VirtIONet *n, VirtIODevice *vdev)
>          }
>      }
>
> -    /*
> -     * Always return true for now until we're able to detect all possible
> -     * changes to a VirtIONet device.
> -     */
> -    return true;
> +    return false;

I'm failing to see one thing: If we add a new feature that the guest
can change during migration and it's only added in the
VMStateDescription vmstate_virtio_net_device, will this "return false"
prevent the state from being resent in the stop-and-copy phase?

Mandating its addition here is ok somehow, but I'm not sure if I'm
missing something.

>  }
>
>  static bool virtio_net_needed(void *opaque)
> diff --git a/include/hw/virtio/virtio-net.h b/include/hw/virtio/virtio-net.h
> index 88074a0976..dbbacc83bb 100644
> --- a/include/hw/virtio/virtio-net.h
> +++ b/include/hw/virtio/virtio-net.h
> @@ -180,6 +180,7 @@ typedef struct VirtIONetQueue {
>   * @guest_offloads_early: Guest offloads snapshot.
>   * @mq_early: Multiqueue state snapshot.
>   * @queue_pairs_early: Queue pairs snapshot.
> + * @tx_waiting_early: Per-queue pending-Tx snapshot.
>   * @rss_enabled_early: RSS enabled flag.
>   * @rss_redirect_early: RSS redirect flag.
>   * @rss_populate_hash_early: RSS populate hash flag.
> @@ -201,6 +202,7 @@ typedef struct VirtIONetMigration {
>      uint64_t guest_offloads_early;
>      int mq_early;
>      uint16_t queue_pairs_early;
> +    uint32_t *tx_waiting_early;
>      bool rss_enabled_early;
>      bool rss_redirect_early;
>      bool rss_populate_hash_early;
> --
> 2.51.0
>

Re: [RFC v2 13/14] virtio-net: detect pending Tx work for VQs mid-migration changes

[Jonah Palmer]

On 3/24/26 7:35 AM, Eugenio Perez Martin wrote:
> On Fri, Mar 20, 2026 at 3:21 PM Jonah Palmer <jonah.palmer@oracle.com> wrote:
>>
>> Track per-queue pending Tx work state during early migration time.
>>
>> This includes a snapshot of pending Tx state for active queue pairs,
>> checking for deltas at the end of migration, and freeing the snapshot
>> buffer during device unrealize.
>>
>> With this final delta signal in place, drop the temporary always-true
>> fallback in virtio_net_has_delta and return false when no deltas are
>> found.
>>
>> Signed-off-by: Jonah Palmer <jonah.palmer@oracle.com>
>> ---
>>   hw/net/virtio-net.c            | 26 +++++++++++++++++++++-----
>>   include/hw/virtio/virtio-net.h |  2 ++
>>   2 files changed, 23 insertions(+), 5 deletions(-)
>>
>> diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
>> index 3ee49a043a..483a43be4f 100644
>> --- a/hw/net/virtio-net.c
>> +++ b/hw/net/virtio-net.c
>> @@ -3943,6 +3943,14 @@ static int virtio_net_early_pre_save(void *opaque)
>>       vnet_mig->mq_early = n->multiqueue;
>>       vnet_mig->queue_pairs_early = n->curr_queue_pairs;
>>
>> +    /* Tx waiting snapshot for active queue pairs */
>> +    if (!vnet_mig->tx_waiting_early) {
>> +        vnet_mig->tx_waiting_early = g_new0(uint32_t, n->max_queue_pairs);
>> +    }
>> +    for (int i = 0; i < n->curr_queue_pairs; i++) {
>> +        vnet_mig->tx_waiting_early[i] = n->vqs[i].tx_waiting;
>> +    }
>> +
>>       /* RSS state snapshot */
>>       vnet_mig->rss_enabled_early = n->rss_data.enabled;
>>       vnet_mig->rss_redirect_early = n->rss_data.redirect;
>> @@ -4254,6 +4262,8 @@ static void virtio_net_device_unrealize(DeviceState *dev)
>>           n->migration->mtable_macs_early = NULL;
>>           g_free(n->migration->vlans_early);
>>           n->migration->vlans_early = NULL;
>> +        g_free(n->migration->tx_waiting_early);
>> +        n->migration->tx_waiting_early = NULL;
>>           g_free(n->migration->rss_indirections_table_early);
>>           n->migration->rss_indirections_table_early = NULL;
>>           g_free(n->migration);
>> @@ -4412,6 +4422,16 @@ static bool virtio_net_has_delta(VirtIONet *n, VirtIODevice *vdev)
>>           return true;
>>       }
>>
>> +    /* Has any active queue's tx_waiting changed? */
>> +    if (!vnet_mig->tx_waiting_early) {
>> +        return true;
>> +    }
>> +    for (int i = 0; i < n->curr_queue_pairs; i++) {
>> +        if (n->vqs[i].tx_waiting != vnet_mig->tx_waiting_early[i]) {
>> +            return true;
>> +        }
>> +    }
>> +
>>       /* Has the VirtIONet's RSS state changed? */
>>       if (n->rss_data.enabled != vnet_mig->rss_enabled_early ||
>>           n->rss_data.redirect != vnet_mig->rss_redirect_early ||
>> @@ -4439,11 +4459,7 @@ static bool virtio_net_has_delta(VirtIONet *n, VirtIODevice *vdev)
>>           }
>>       }
>>
>> -    /*
>> -     * Always return true for now until we're able to detect all possible
>> -     * changes to a VirtIONet device.
>> -     */
>> -    return true;
>> +    return false;
> 
> I'm failing to see one thing: If we add a new feature that the guest
> can change during migration and it's only added in the
> VMStateDescription vmstate_virtio_net_device, will this "return false"
> prevent the state from being resent in the stop-and-copy phase?
> 
> Mandating its addition here is ok somehow, but I'm not sure if I'm
> missing something.
> 

Correct. But this is also the same case today. That is, any new 
migratable device state would still need to be added to the appropriate 
VMSD/subsection. For example, if today we added a new member to 
VirtIONet that also should be migrated, it would also need to be added 
to vmstate_virtio_net_device (or the appropriate subsection).

So it's not a new *kind* of maintenance burden, but it is an additional 
place that'd need to be updated.

>>   }
>>
>>   static bool virtio_net_needed(void *opaque)
>> diff --git a/include/hw/virtio/virtio-net.h b/include/hw/virtio/virtio-net.h
>> index 88074a0976..dbbacc83bb 100644
>> --- a/include/hw/virtio/virtio-net.h
>> +++ b/include/hw/virtio/virtio-net.h
>> @@ -180,6 +180,7 @@ typedef struct VirtIONetQueue {
>>    * @guest_offloads_early: Guest offloads snapshot.
>>    * @mq_early: Multiqueue state snapshot.
>>    * @queue_pairs_early: Queue pairs snapshot.
>> + * @tx_waiting_early: Per-queue pending-Tx snapshot.
>>    * @rss_enabled_early: RSS enabled flag.
>>    * @rss_redirect_early: RSS redirect flag.
>>    * @rss_populate_hash_early: RSS populate hash flag.
>> @@ -201,6 +202,7 @@ typedef struct VirtIONetMigration {
>>       uint64_t guest_offloads_early;
>>       int mq_early;
>>       uint16_t queue_pairs_early;
>> +    uint32_t *tx_waiting_early;
>>       bool rss_enabled_early;
>>       bool rss_redirect_early;
>>       bool rss_populate_hash_early;
>> --
>> 2.51.0
>>
> 

Re: [RFC v2 13/14] virtio-net: detect pending Tx work for VQs mid-migration changes

[Eugenio Perez Martin]

On Tue, Mar 24, 2026 at 5:48 PM Jonah Palmer <jonah.palmer@oracle.com> wrote:
>
>
>
> On 3/24/26 7:35 AM, Eugenio Perez Martin wrote:
> > On Fri, Mar 20, 2026 at 3:21 PM Jonah Palmer <jonah.palmer@oracle.com> wrote:
> >>
> >> Track per-queue pending Tx work state during early migration time.
> >>
> >> This includes a snapshot of pending Tx state for active queue pairs,
> >> checking for deltas at the end of migration, and freeing the snapshot
> >> buffer during device unrealize.
> >>
> >> With this final delta signal in place, drop the temporary always-true
> >> fallback in virtio_net_has_delta and return false when no deltas are
> >> found.
> >>
> >> Signed-off-by: Jonah Palmer <jonah.palmer@oracle.com>
> >> ---
> >>   hw/net/virtio-net.c            | 26 +++++++++++++++++++++-----
> >>   include/hw/virtio/virtio-net.h |  2 ++
> >>   2 files changed, 23 insertions(+), 5 deletions(-)
> >>
> >> diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
> >> index 3ee49a043a..483a43be4f 100644
> >> --- a/hw/net/virtio-net.c
> >> +++ b/hw/net/virtio-net.c
> >> @@ -3943,6 +3943,14 @@ static int virtio_net_early_pre_save(void *opaque)
> >>       vnet_mig->mq_early = n->multiqueue;
> >>       vnet_mig->queue_pairs_early = n->curr_queue_pairs;
> >>
> >> +    /* Tx waiting snapshot for active queue pairs */
> >> +    if (!vnet_mig->tx_waiting_early) {
> >> +        vnet_mig->tx_waiting_early = g_new0(uint32_t, n->max_queue_pairs);
> >> +    }
> >> +    for (int i = 0; i < n->curr_queue_pairs; i++) {
> >> +        vnet_mig->tx_waiting_early[i] = n->vqs[i].tx_waiting;
> >> +    }
> >> +
> >>       /* RSS state snapshot */
> >>       vnet_mig->rss_enabled_early = n->rss_data.enabled;
> >>       vnet_mig->rss_redirect_early = n->rss_data.redirect;
> >> @@ -4254,6 +4262,8 @@ static void virtio_net_device_unrealize(DeviceState *dev)
> >>           n->migration->mtable_macs_early = NULL;
> >>           g_free(n->migration->vlans_early);
> >>           n->migration->vlans_early = NULL;
> >> +        g_free(n->migration->tx_waiting_early);
> >> +        n->migration->tx_waiting_early = NULL;
> >>           g_free(n->migration->rss_indirections_table_early);
> >>           n->migration->rss_indirections_table_early = NULL;
> >>           g_free(n->migration);
> >> @@ -4412,6 +4422,16 @@ static bool virtio_net_has_delta(VirtIONet *n, VirtIODevice *vdev)
> >>           return true;
> >>       }
> >>
> >> +    /* Has any active queue's tx_waiting changed? */
> >> +    if (!vnet_mig->tx_waiting_early) {
> >> +        return true;
> >> +    }
> >> +    for (int i = 0; i < n->curr_queue_pairs; i++) {
> >> +        if (n->vqs[i].tx_waiting != vnet_mig->tx_waiting_early[i]) {
> >> +            return true;
> >> +        }
> >> +    }
> >> +
> >>       /* Has the VirtIONet's RSS state changed? */
> >>       if (n->rss_data.enabled != vnet_mig->rss_enabled_early ||
> >>           n->rss_data.redirect != vnet_mig->rss_redirect_early ||
> >> @@ -4439,11 +4459,7 @@ static bool virtio_net_has_delta(VirtIONet *n, VirtIODevice *vdev)
> >>           }
> >>       }
> >>
> >> -    /*
> >> -     * Always return true for now until we're able to detect all possible
> >> -     * changes to a VirtIONet device.
> >> -     */
> >> -    return true;
> >> +    return false;
> >
> > I'm failing to see one thing: If we add a new feature that the guest
> > can change during migration and it's only added in the
> > VMStateDescription vmstate_virtio_net_device, will this "return false"
> > prevent the state from being resent in the stop-and-copy phase?
> >
> > Mandating its addition here is ok somehow, but I'm not sure if I'm
> > missing something.
> >
>
> Correct. But this is also the same case today. That is, any new
> migratable device state would still need to be added to the appropriate
> VMSD/subsection. For example, if today we added a new member to
> VirtIONet that also should be migrated, it would also need to be added
> to vmstate_virtio_net_device (or the appropriate subsection).
>
> So it's not a new *kind* of maintenance burden, but it is an additional
> place that'd need to be updated.
>

Right, we're on the same page.

Half-baked idea: Is it possible to update it only in
vmstate_virtio_net_device? I'm thinking in a method to store only the
relevant fields by iterating vmstate_virtio_net_device, instead of
manually coding each if (n->field != vnet_mig->field) return true.

> >>   }
> >>
> >>   static bool virtio_net_needed(void *opaque)
> >> diff --git a/include/hw/virtio/virtio-net.h b/include/hw/virtio/virtio-net.h
> >> index 88074a0976..dbbacc83bb 100644
> >> --- a/include/hw/virtio/virtio-net.h
> >> +++ b/include/hw/virtio/virtio-net.h
> >> @@ -180,6 +180,7 @@ typedef struct VirtIONetQueue {
> >>    * @guest_offloads_early: Guest offloads snapshot.
> >>    * @mq_early: Multiqueue state snapshot.
> >>    * @queue_pairs_early: Queue pairs snapshot.
> >> + * @tx_waiting_early: Per-queue pending-Tx snapshot.
> >>    * @rss_enabled_early: RSS enabled flag.
> >>    * @rss_redirect_early: RSS redirect flag.
> >>    * @rss_populate_hash_early: RSS populate hash flag.
> >> @@ -201,6 +202,7 @@ typedef struct VirtIONetMigration {
> >>       uint64_t guest_offloads_early;
> >>       int mq_early;
> >>       uint16_t queue_pairs_early;
> >> +    uint32_t *tx_waiting_early;
> >>       bool rss_enabled_early;
> >>       bool rss_redirect_early;
> >>       bool rss_populate_hash_early;
> >> --
> >> 2.51.0
> >>
> >
>

[RFC v2 14/14] virtio-net, vhost-net: early migration support for vhost-net

[Jonah Palmer via qemu development]

This patch implements early migration support for virtio-net devices
using a TAP backend accelerated by vhost-net.

More specifically, we initiate the vhost startup routine during early
migration time but guard against binding TAP backends at this time. We
need to wait until the source VM has been paused before we want the
device to actually start.

For vhost-net, the remaining stop-and-copy work is to apply final vring
bases and bind TAP backends. This is handled via the virtio-net vhost
subsection (vmstate_virtio_net_vhost) post_load callback.

When a mid-migration delta is detected and we fall back to a full
virtio-net reload, explicitly stop any early-started vhost instance
before restart so notifier/backend state is handled safely.

Failures while starting vhost-net during early post-load, and failures
during stop-and-copy quickstart finalization, are treated as non-fatal
for migration. In those cases the destination continues migration and
falls back to userspace virtio-net datapath. After switchover, the
normal vhost start path may retry once status is set; if that retry also
fails, the device continues running on userspace virtio-net.

By moving most of the post-load startup work out of the stop-and-copy
phase, we further minimize the guest-visible downtime incurred by
migrating a virtio-net device using vhost-net.

A future improvement to this patch should handle deltas more gracefully
by updating only what was changed mid-migration instead of relying on a
full vhost/virtio-net restart.

Signed-off-by: Jonah Palmer <jonah.palmer@oracle.com>
---
 hw/net/vhost_net.c             | 183 +++++++++++++++++++++++++++++++++
 hw/net/virtio-net.c            | 127 ++++++++++++++++++++++-
 include/hw/virtio/virtio-net.h |   2 +
 include/net/vhost_net.h        |   9 ++
 4 files changed, 319 insertions(+), 2 deletions(-)

diff --git a/hw/net/vhost_net.c b/hw/net/vhost_net.c
index a8ee18a912..f11f30b4f0 100644
--- a/hw/net/vhost_net.c
+++ b/hw/net/vhost_net.c
@@ -353,6 +353,13 @@ static int vhost_net_start_one(struct vhost_net *net,
                 /* Queue might not be ready for start */
                 continue;
             }
+            if (dev->migration && dev->migration->early_load) {
+                /*
+                 * Queue isn't ready to start as we're in the middle of an
+                 * early migration. Set the backend later when we're ready.
+                 */
+                continue;
+            }
             r = vhost_net_set_backend(&net->dev, &file);
             if (r < 0) {
                 r = -errno;
@@ -695,3 +702,179 @@ err_start:
 
     return r;
 }
+
+/*
+ * Helper function for vhost_net_post_load_migration_quickstart:
+ *
+ * Sets vring bases for all vhost virtqueues.
+ */
+int vhost_net_set_all_vring_bases(struct VirtIONet *n, VirtIODevice *vdev,
+                                  NetClientState *ncs, int queue_pairs,
+                                  int cvq, int nvhosts)
+{
+    NetClientState *peer;
+    struct vhost_net *vnet;
+    struct vhost_dev *hdev;
+    int queue_idx;
+    int i, j, r;
+
+    for (i = 0; i < nvhosts; i++) {
+        peer = qemu_get_peer(ncs, i < queue_pairs ? i : n->max_queue_pairs);
+        vnet = get_vhost_net(peer);
+        if (!vnet) {
+            continue;
+        }
+        hdev = &vnet->dev;
+
+        for (j = 0; j < hdev->nvqs; ++j) {
+            queue_idx = hdev->vq_index + j;
+            struct vhost_vring_state state = {
+                .index = hdev->vhost_ops->vhost_get_vq_index(hdev, queue_idx),
+                .num = virtio_queue_get_last_avail_idx(vdev, queue_idx),
+            };
+
+            r = hdev->vhost_ops->vhost_set_vring_base(hdev, &state);
+            if (r) {
+                error_report("vhost_set_vring_base failed (vq %d)", queue_idx);
+                goto fail;
+            }
+        }
+    }
+    return 0;
+
+fail:
+    vhost_net_stop_one(vnet, vdev);
+
+    while (--i >= 0) {
+        peer = qemu_get_peer(ncs, i < queue_pairs ? i : n->max_queue_pairs);
+        vhost_net_stop_one(get_vhost_net(peer), vdev);
+    }
+    return r;
+}
+
+/*
+ * Helper function for vhost_net_post_load_migration_quickstart:
+ *
+ * Binds TAP backends to all vhost-net virtqueues. All vring bases must be set
+ * before attempting to start any backends.
+ */
+int vhost_net_start_all_backends(struct VirtIONet *n, VirtIODevice *vdev,
+                                 NetClientState *ncs, int queue_pairs, int cvq,
+                                 int nvhosts)
+{
+    NetClientState *peer;
+    struct vhost_dev *hdev;
+    struct vhost_vring_file file = { };
+    struct vhost_net *vnet;
+    int i, r;
+
+    for (i = 0; i < nvhosts; i++) {
+        peer = qemu_get_peer(ncs, i < queue_pairs ? i : n->max_queue_pairs);
+        vnet = get_vhost_net(peer);
+        if (!vnet) {
+            continue;
+        }
+        hdev = &vnet->dev;
+
+        qemu_set_fd_handler(vnet->backend, NULL, NULL, NULL);
+        file.fd = vnet->backend;
+
+        for (file.index = 0; file.index < hdev->nvqs; ++file.index) {
+            if (!virtio_queue_enabled(vdev, hdev->vq_index + file.index)) {
+                /* Queue might not be ready to start */
+                continue;
+            }
+
+            r = vhost_net_set_backend(hdev, &file);
+            if (r < 0) {
+                r = -errno;
+                goto fail;
+            }
+        }
+    }
+    return 0;
+
+fail:
+    file.fd = -1;
+    while (file.index-- > 0) {
+        if (!virtio_queue_enabled(vdev, hdev->vq_index + file.index)) {
+            continue;
+        }
+        int ret = vhost_net_set_backend(hdev, &file);
+        assert(ret >= 0);
+    }
+    if (vnet->nc->info->poll) {
+        vnet->nc->info->poll(vnet->nc, true);
+    }
+    vhost_dev_stop(hdev, vdev, false);
+
+    while (--i >= 0) {
+        peer = qemu_get_peer(ncs, i < queue_pairs ? i : n->max_queue_pairs);
+        vhost_net_stop_one(get_vhost_net(peer), vdev);
+    }
+    return r;
+}
+
+/*
+ * Quickstart path for a virtio-net device using vhost acceleration:
+ *
+ * Used during migration of a virtio-net device that opted-in to early
+ * migration.
+ *
+ * The goal of this function is to perform any remaining startup work that
+ * can only be done during the stop-and-copy phase, once the source has been
+ * stopped.
+ *
+ * Note: By the time this function is called, the device has essentially been
+ * fully configured, albeit with a few last-minute configurations to be made.
+ * This means our error handling must completely unwind the device with
+ * full-stop semantics.
+ */
+int vhost_net_post_load_migration_quickstart(struct VirtIONet *n)
+{
+    VirtIODevice *vdev = VIRTIO_DEVICE(n);
+    NetClientState *ncs = qemu_get_queue(n->nic);
+    BusState *qbus = BUS(qdev_get_parent_bus(DEVICE(vdev)));
+    VirtioBusState *vbus = VIRTIO_BUS(qbus);
+    VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(vbus);
+
+    int queue_pairs = n->multiqueue ? n->max_queue_pairs : 1;
+    int cvq = virtio_vdev_has_feature(vdev, VIRTIO_NET_F_CTRL_VQ) ?
+              n->max_ncs - n->max_queue_pairs : 0;
+    int nvhosts = queue_pairs + cvq;
+    int total_notifiers = queue_pairs * 2 + cvq;
+    NetClientState *peer = qemu_get_peer(ncs, 0);
+
+    int r, e;
+
+    /* First peer must exist for the realized virtio-net device */
+    assert(peer);
+
+    /* Apply final vring bases for all vhosts */
+    r = vhost_net_set_all_vring_bases(n, vdev, ncs, queue_pairs, cvq, nvhosts);
+    if (r < 0) {
+        goto fail;
+    }
+
+    /* Bind backends (TAP devices only) */
+    if (peer->info->type == NET_CLIENT_DRIVER_TAP) {
+        r = vhost_net_start_all_backends(n, vdev, ncs, queue_pairs, cvq, nvhosts);
+        if (r < 0) {
+            goto fail;
+        }
+    }
+    return 0;
+
+fail:
+    e = k->set_guest_notifiers(qbus->parent, total_notifiers, false);
+    if (e < 0) {
+        fprintf(stderr, "vhost guest notifier cleanup failed: %d\n", e);
+        fflush(stderr);
+    }
+    vhost_net_disable_notifiers(vdev, ncs, queue_pairs, cvq);
+
+    error_report("unable to start vhost net: %d: "
+                 "falling back on userspace virtio", -r);
+    n->vhost_started = 0;
+    return r;
+}
diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index 483a43be4f..950137c568 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -3864,6 +3864,38 @@ static bool failover_hide_primary_device(DeviceListener *listener,
     return qatomic_read(&n->failover_primary_hidden);
 }
 
+static int virtio_net_vhost_early_start(VirtIONet *n, VirtIODevice *vdev)
+{
+    NetClientState *ncs = qemu_get_queue(n->nic);
+    int queue_pairs = n->multiqueue ? n->max_queue_pairs : 1;
+    int cvq = virtio_vdev_has_feature(vdev, VIRTIO_NET_F_CTRL_VQ) ?
+              n->max_ncs - n->max_queue_pairs : 0;
+    int r;
+
+    /* Return early if there's no vhost backend */
+    if (!ncs || !ncs->peer || !get_vhost_net(ncs->peer)) {
+        return 0;
+    }
+
+    if (virtio_has_feature(vdev->guest_features, VIRTIO_NET_F_MTU)) {
+        r = vhost_net_set_mtu(get_vhost_net(ncs->peer), n->net_conf.mtu);
+        if (r < 0) {
+            error_report("%u bytes MTU not supported by the backend",
+                         n->net_conf.mtu);
+            return r;
+        }
+    }
+
+    n->vhost_started = 1;
+    r = vhost_net_start(vdev, n->nic->ncs, queue_pairs, cvq);
+    if (r < 0) {
+        error_report("unable to start vhost net: %d: "
+                     "falling back on userspace virtio", -r);
+        n->vhost_started = 0;
+    }
+    return r;
+}
+
 enum VirtIONetRxFlags {
     VNET_RX_F_PROMISC  = 1u << 0,
     VNET_RX_F_ALLMULTI = 1u << 1,
@@ -3892,6 +3924,9 @@ static int virtio_net_early_pre_save(void *opaque)
     VirtIONetMigration *vnet_mig = n->migration;
     size_t vlans_size = (size_t)(MAX_VLAN >> 3);
 
+    /* Reset source-side delta decision for this migration iteration. */
+    n->migration->reloaded = false;
+
     vdev_mig->status_early = vdev->status;
     vnet_mig->status_early = n->status;
 
@@ -3989,6 +4024,14 @@ static int virtio_net_early_post_load(void *opaque, int version_id)
     VirtIONet *n = opaque;
     VirtIODevice *vdev = VIRTIO_DEVICE(n);
 
+    /*
+     * Start the vhost backend if one is present. Note that while
+     * vdev->migration->early_load is true, not all vhost startup operations
+     * are performed. For example, we defer setting the backends (vhost-net w/
+     * TAP) until the stop-and-copy phase (see vmstate_virtio_net_vhost).
+     */
+    virtio_net_vhost_early_start(n, vdev);
+
     vdev->migration->early_load = false;
     return 0;
 }
@@ -4007,6 +4050,49 @@ static const VMStateDescription vmstate_virtio_net_early = {
     },
 };
 
+static int virtio_net_vhost_post_load(void *opaque, int version_id)
+{
+    VirtIONet *n = opaque;
+    int r;
+
+    if (!n->vhost_started) {
+        return 0;
+    }
+
+    /* Finalize vhost startup */
+    r = vhost_net_post_load_migration_quickstart(n);
+    if (r < 0) {
+        error_report("virtio-net vhost post-load quickstart failed: %d", r);
+    }
+    return 0;
+}
+
+static bool virtio_net_vhost_needed(void *opaque)
+{
+    VirtIONet *n = opaque;
+    NetClientState *nc = qemu_get_queue(n->nic);
+
+    if (!nc || !nc->peer || !get_vhost_net(nc->peer)) {
+        return false;
+    }
+
+    /* Skip vhost quickstart section when a full virtio-net reload is needed. */
+    return !n->migration->reloaded;
+}
+
+static const VMStateDescription vmstate_virtio_net_vhost = {
+    .name = "virtio-net-vhost",
+    .minimum_version_id = 1,
+    .version_id = 1,
+    /* Set prio low to run after vmstate_virtio_net */
+    .priority = MIG_PRI_LOW,
+    .needed = virtio_net_vhost_needed,
+    .fields = (const VMStateField[]) {
+        VMSTATE_END_OF_LIST()
+    },
+    .post_load = virtio_net_vhost_post_load,
+};
+
 static void virtio_net_device_realize(DeviceState *dev, Error **errp)
 {
     VirtIODevice *vdev = VIRTIO_DEVICE(dev);
@@ -4201,9 +4287,10 @@ static void virtio_net_device_realize(DeviceState *dev, Error **errp)
             vdev->migration = g_new0(VirtIODevMigration, 1);
             vdev->migration->early_load = false;
             n->migration = g_new0(VirtIONetMigration, 1);
-
             vmstate_register_any(VMSTATE_IF(n), &vmstate_virtio_net_early, n);
             virtio_delta_vmsd_register(vdev);
+            vmstate_register_any(VMSTATE_IF(n), &vmstate_virtio_net_vhost,
+                                 n);
         }
     }
 }
@@ -4271,6 +4358,7 @@ static void virtio_net_device_unrealize(DeviceState *dev)
 
         vmstate_unregister(VMSTATE_IF(n), &vmstate_virtio_net_early, n);
         virtio_delta_vmsd_unregister(vdev);
+        vmstate_unregister(VMSTATE_IF(n), &vmstate_virtio_net_vhost, n);
     }
 }
 
@@ -4336,6 +4424,37 @@ static int virtio_net_pre_save(void *opaque)
     return 0;
 }
 
+static int virtio_net_pre_load(void *opaque)
+{
+    VirtIONet *n = opaque;
+    VirtIODevice *vdev = VIRTIO_DEVICE(n);
+
+    /*
+     * If we're migrating with a vhost device and performed an early
+     * save/load, then reaching here means that something changed and
+     * we need to reload all of the virtio-net device's state.
+     */
+    if (n->early_mig) {
+        /*
+         * Unwind vhost-net before full reload path re-runs startup. This keeps
+         * notifier/backend state handling safe.
+         */
+        if (n->vhost_started) {
+            NetClientState *nc = qemu_get_queue(n->nic);
+            int queue_pairs = n->multiqueue ? n->max_queue_pairs : 1;
+            int cvq = virtio_vdev_has_feature(vdev, VIRTIO_NET_F_CTRL_VQ) ?
+                      n->max_ncs - n->max_queue_pairs : 0;
+
+            if (nc && nc->peer && get_vhost_net(nc->peer)) {
+                vhost_net_stop(vdev, n->nic->ncs, queue_pairs, cvq);
+            }
+
+            n->vhost_started = 0;
+        }
+    }
+    return 0;
+}
+
 static bool primary_unplug_pending(void *opaque)
 {
     DeviceState *dev = opaque;
@@ -4466,12 +4585,15 @@ static bool virtio_net_needed(void *opaque)
 {
     VirtIONet *n = opaque;
     VirtIODevice *vdev = VIRTIO_DEVICE(n);
+    bool delta;
 
     if (!n->early_mig) {
         return true;
     }
 
-    return virtio_net_has_delta(n, vdev);
+    delta = virtio_net_has_delta(n, vdev);
+    n->migration->reloaded = delta;
+    return delta;
 }
 
 static const VMStateDescription vmstate_virtio_net = {
@@ -4484,6 +4606,7 @@ static const VMStateDescription vmstate_virtio_net = {
         VMSTATE_END_OF_LIST()
     },
     .pre_save = virtio_net_pre_save,
+    .pre_load = virtio_net_pre_load,
     .dev_unplug_pending = dev_unplug_pending,
 };
 
diff --git a/include/hw/virtio/virtio-net.h b/include/hw/virtio/virtio-net.h
index dbbacc83bb..d1d7c0b742 100644
--- a/include/hw/virtio/virtio-net.h
+++ b/include/hw/virtio/virtio-net.h
@@ -169,6 +169,7 @@ typedef struct VirtIONetQueue {
 
 /**
  * struct VirtIONetMigration - VirtIONet migration structure
+ * @reloaded: Flag to indicate the state has been reloaded.
  * @status_early: VirtIONet status snapshot.
  * @mac_early: MAC address early migration snapshot.
  * @mtable_in_use_early: In-use MAC table entries.
@@ -191,6 +192,7 @@ typedef struct VirtIONetQueue {
  * @rss_indirections_table_early: RSS indirections table.
  */
 typedef struct VirtIONetMigration {
+    bool reloaded;
     uint16_t status_early;
     uint8_t mac_early[ETH_ALEN];
     uint32_t mtable_in_use_early;
diff --git a/include/net/vhost_net.h b/include/net/vhost_net.h
index 0225207491..a8a1c1005b 100644
--- a/include/net/vhost_net.h
+++ b/include/net/vhost_net.h
@@ -4,6 +4,7 @@
 #include "net/net.h"
 #include "hw/virtio/virtio-features.h"
 #include "hw/virtio/vhost-backend.h"
+#include "hw/virtio/virtio-net.h"
 
 struct vhost_net;
 typedef struct vhost_net VHostNetState;
@@ -88,4 +89,12 @@ int vhost_net_virtqueue_restart(VirtIODevice *vdev, NetClientState *nc,
                                 int vq_index);
 
 void vhost_net_save_acked_features(NetClientState *nc);
+
+int vhost_net_set_all_vring_bases(struct VirtIONet *n, VirtIODevice *vdev,
+                                  NetClientState *ncs, int queue_pairs,
+                                  int cvq, int nvhosts);
+int vhost_net_start_all_backends(struct VirtIONet *n, VirtIODevice *vdev,
+                                  NetClientState *ncs, int queue_pairs,
+                                  int cvq, int nvhosts);
+int vhost_net_post_load_migration_quickstart(struct VirtIONet *n);
 #endif
-- 
2.51.0