[RFC v2 00/14] virtio-net: early VMStateDescription live migration support

[Jonah Palmer <jonah.palmer@oracle.com>]

This RFC explores using VMStateDescription to move virtio-net migration
work out of stop-and-copy. The goal is to reduce guest-visible downtime
by loading as much virtio-net state as possible during the setup phase,
then deciding at stop-and-copy whether the destination can finish with a
minimal resync or needs to fall back to the existing full reload path.

This is a continuation of the earlier RFC that prototyped the idea with
SaveVMHandlers [1]. This version reworks the approach around
VMStateDescription.

At a high level, the series does the following:
- adds an `early-mig` property for virtio-net and enables it by default
  only for newer machine types, while older machine types keep the
  legacy default through compat properties
- adds an early_setup-based VMSD so virtio-net setup work can be moved
  before stop-and-copy
- snapshots state that was sent early, detects relevant mid-migration
  deltas at stop-and-copy, and falls back to the existing full
  virtio-net reload path if any tracked state changed
- if no tracked deltas are found, skips the full reload and only resends
  the minimal state that still must be synchronized at stop-and-copy
- extends the same model to vhost-net by starting most vhost state early
  and deferring backend binding/finalization until stop-and-copy

This RFC series works as follows:
- state loaded early is treated as provisional
- at stop-and-copy, tracked VirtIODevice/VirtIONet state is compared
  against the early snapshot
- if anything relevant changed, migration falls back to the existing
  full virtio-net reload path
- only the no-delta case keeps the shorter stop-and-copy path

For an actual patch series, the following should be implemented:
- handle deltas individually instead of forcing a full reload whenever
  any tracked state changes
- include support for vhost-vDPA

The `early-mig` property is enabled by default only for machine types >=
10.2. Older machine types keep the legacy default off through compat
properties, so existing machine-version behavior is preserved unless the
user overrides the setting explicitly.

For vhost-net, failures while starting vhost early or during the
stop-and-copy quickstart path are treated as non-fatal to migration. In
those cases the destination continues on the userspace virtio-net
datapath, and the normal post-switchover vhost start path may retry
later. If a delta forces a full reload, any early-started vhost
instance is stopped before restart so notifier/backend state is torn
down safely.

Testing framework:
- Local live migration testing compared:
  - baseline: the current VMSD migration flow
  - early VMSD: this RFC series
- Setup:
  - 30 measured runs after 3 warmup runs per mode (baseline vs early)
  - virtio-net with 4 queue pairs
  - guest UDP request/response probe sampled at 10 ms intervals
  - no configuration deltas injected; only early VMSD path used
- Metrics:
  - query-migrate downtime
  - maximum observed probe gap as a guest-visible interruption proxy

Testing results:
- vhost=on (n=30):
  - query-migrate downtime median: 69.0 ms baseline, 60.5 ms early
    (12.3% lower)
  - maximum probe gap median: 309.0 ms baseline, 282.5 ms early
    (8.6% lower)
  - p95: downtime 85 ms baseline vs 86 ms early; probe gap 338 ms in
    both modes

- vhost=off (n=30):
  - `query-migrate` downtime median: 55.0 ms baseline, 49.0 ms early
    (10.9% lower)
  - maximum probe gap median: 283.5 ms baseline, 252.5 ms early
    (10.9% lower)
  - p95: downtime 69 ms baseline vs 81 ms early; probe gap 310.0 ms
    baseline vs 301.0 ms early

Median results improved relative to baseline in both the vhost=on and
vhost=off runs. Higher-percentile results were mixed, especially for
query-migrate downtime in the vhost=off case. That may partly reflect
the relatively small test setup and the fact that these are broader
migration-level or end-to-end measurements rather than explicit
virtio-net device downtime.

A non-RFC submission will have more targeted testing with larger
migration state and more explicit device-focused measurement.

Comments on the overall VMSD structure, migration compatibility
implications, and whether this is the right direction for early
virtio-net device migration would be appreciated.

[1] https://lore.kernel.org/qemu-devel/20250722124127.2497406-1-jonah.palmer@oracle.com/

Jonah Palmer (14):
  machine,virtio-net: add early-mig property
  virtio,virtio-net: add initial early VMSD for setup-phase migration
  virtio,virtio-net: virtio-delta VMSD - VQ state
  virtio-net: detect VirtIODevice status mid-migration change
  virtio-net: detect VirtIODevice config buffer mid-migration change
  virtio-net: detect VirtIONet MAC addr mid-migration change
  virtio-net: detect VirtIONet MAC table mid-migration changes
  virtio-net: detect VirtIONet status mid-migration change
  virtio-net: detect VirtIONet Rx filter mid-migration changes
  virtio-net: detect VirtIONet VLAN filter table changes
  virtio-net: detect VirtIONet guest offload & MQ mid-migration changes
  virtio-net: detect RSS state mid-migration changes
  virtio-net: detect pending Tx work for VQs mid-migration changes
  virtio-net,vhost-net: early migration support for vhost-net

 hw/core/machine.c              |   5 +
 hw/net/vhost_net.c             | 183 ++++++++++++++
 hw/net/virtio-net.c            | 423 +++++++++++++++++++++++++++++++++
 hw/virtio/virtio.c             | 164 ++++++++++++-
 include/hw/virtio/virtio-net.h |  50 ++++
 include/hw/virtio/virtio.h     |  18 ++
 include/net/vhost_net.h        |   9 +
 7 files changed, 851 insertions(+), 1 deletion(-)

-- 
2.51.0