Re: [PATCH v16 5/8] virtio-net: support local migration of backend

[Peter Xu <peterx@redhat.com>]

On Mon, May 25, 2026 at 04:51:55PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> On 24.05.26 12:09, Michael S. Tsirkin wrote:
> > On Fri, May 22, 2026 at 03:05:30PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> > > Add virtio-net option local-migration, which is true by default,
> > > but false for older machine types, which doesn't support the feature.
> > > 
> > > When both global migration parameter "local" and new virtio-net
> > > parameter "local-migration" are true, virtio-net transfer the whole
> > > net backend to the destination, including open file descriptors.
> > > Of-course, its only for local migration and the channel must be
> > > UNIX domain socket.
> > > 
> > > This way management tool should not care about creating new TAP, and
> > > should not handle switching to it. Migration downtime become shorter.
> > > 
> > > Support for TAP will come in the next commit.
> > > 
> > > Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
> > > Reviewed-by: Ben Chaney <bchaney@akamai.com>
> > 
> > I don't get why is this a device property?
> > It's clearly a backend thing?
> > 
> 
> Hmm.
> 
> We want to be able to disable fd-migration per device, when common "local"
> migration parameter is on.
> 
> That's why we have parameter for virtio-net. It's also good, that we may
> enable it by default for newer machine types, to make fd-migration a default
> path.
> 
> To be honest, it seems that the only thing in this patch (except for interface),
> which is not about backand, is that we want to do virtio_net_update_host_features()
> after backends incoming migration finished. For this, it's comfortable to have
> backend migration as part of device migration stream.
> 
> --
> 
> Imagine, we move "local-migration" parameter to TAP device. This raise a lot of questions:

Hmm, I thought we discussed this quite some time ago..  I can't remember
details, but I'll try to comment with what I can still remember.

> 
> 1. We loss a possibility to set good default in machine type. Ok, we probably may add
> a property to "migration" object, but this property will look like "TAP-local-migration",
> and raise discussion again, that it should be property of TAP..

If you recall I worked on the other series because of this desire of having
tap be able to be QOMified and also support machine compat properties:

https://lore.kernel.org/all/20251209162857.857593-2-peterx@redhat.com/

I didn't get a lot of feedback supporting having TYPE_OBJECT_COMPAT, maybe
it's an overkill only to apply object_apply_compat_props().

However, just to say we can still QOMify TAP and then add its own
instance_post_init() to also do object_apply_compat_props(), like quite a
few other existing users, then this (1) isn't a problem, and it doesn't
need to depend on my series either.

> 
> 2. We loss possiblity to use qom-set, to change the property value, as net backends are
> not Qobjects.. So it will need additional tap-change qmp command or something like this.

This also shouldn't be a problem if we QOMify TAP, IIUC.

> 
> 3. Migration it self: should we register specific migration state for TAP backend (which is
> not very comfortable, because net backends are not devices), or just keep it as part of
> virtio-net migration state (like in this patch, but instead of checking for own local-migration
> property, check for all backends local-migration properties).
> 
> If create personal migration state for TAP backend, detached from virtio-net migration state,
> we'll need somehow call virtio_net_update_host_features() _after_ all backends migrated. May
> be we can do it in device start, but it's not good, as at this point any error is more critical
> than during migration process.. Another option is migration state priorities, but I didn't dig
> into it, and it doesn't seem as reliable as simply migrate backends as part of parent device
> migration state.

If this is about the "which VMSD to migrate earlier" problem, I recall we
discussed about using MigrationPriority to serialize them.  But I'm not
sure if this is exactly the same issue.

Thanks,

> 
> --
> 
> What do you think? Theoretically, I can move "local-migration" property to TAP backend, but
> seems, it will lead to more problems.
> 
> > 
> > 
> > > ---
> > >   hw/core/machine.c              |   1 +
> > >   hw/i386/pc_q35.c               |   1 +
> > >   hw/net/virtio-net.c            | 137 ++++++++++++++++++++++++++++++++-
> > >   include/hw/virtio/virtio-net.h |   2 +
> > >   include/net/net.h              |   2 +
> > >   5 files changed, 142 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/hw/core/machine.c b/hw/core/machine.c
> > > index 63baff859f3..619e80c1cb3 100644
> > > --- a/hw/core/machine.c
> > > +++ b/hw/core/machine.c
> > > @@ -41,6 +41,7 @@
> > >   GlobalProperty hw_compat_11_0[] = {
> > >       { "chardev-vc", "encoding", "cp437" },
> > > +    { TYPE_VIRTIO_NET, "local-migration", "false" },
> > >   };
> > >   const size_t hw_compat_11_0_len = G_N_ELEMENTS(hw_compat_11_0);
> > > diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
> > > index d8fed698c72..b5c0e302d59 100644
> > > --- a/hw/i386/pc_q35.c
> > > +++ b/hw/i386/pc_q35.c
> > > @@ -368,6 +368,7 @@ static void pc_q35_machine_options(MachineClass *m)
> > >   static void pc_q35_machine_11_1_options(MachineClass *m)
> > >   {
> > >       pc_q35_machine_options(m);
> > > +    compat_props_add(m->compat_props, hw_compat_11_0, hw_compat_11_0_len);
> > >   }
> > >   DEFINE_Q35_MACHINE_AS_LATEST(11, 1);
> > > diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
> > > index 2a5d642a647..158b9247a58 100644
> > > --- a/hw/net/virtio-net.c
> > > +++ b/hw/net/virtio-net.c
> > > @@ -38,8 +38,10 @@
> > >   #include "qapi/qapi-events-migration.h"
> > >   #include "hw/virtio/virtio-access.h"
> > >   #include "migration/misc.h"
> > > +#include "migration/options.h"
> > >   #include "standard-headers/linux/ethtool.h"
> > >   #include "system/system.h"
> > > +#include "system/runstate.h"
> > >   #include "system/replay.h"
> > >   #include "trace.h"
> > >   #include "monitor/qdev.h"
> > > @@ -3060,7 +3062,17 @@ static void virtio_net_set_multiqueue(VirtIONet *n, int multiqueue)
> > >       n->multiqueue = multiqueue;
> > >       virtio_net_change_num_queues(n, max * 2 + 1);
> > > -    virtio_net_set_queue_pairs(n);
> > > +    /*
> > > +     * virtio_net_set_multiqueue() called from set_features(0) on early
> > > +     * reset, when peer may wait for incoming (and is not initialized
> > > +     * yet).
> > > +     * Don't worry about it: virtio_net_set_queue_pairs() will be called
> > > +     * later form virtio_net_post_load_device(), and anyway will be
> > > +     * noop for local incoming migration with live backend passing.
> > > +     */
> > > +    if (!n->peers_wait_incoming) {
> > > +        virtio_net_set_queue_pairs(n);
> > > +    }
> > >   }
> > >   static int virtio_net_pre_load_queues(VirtIODevice *vdev, uint32_t n)
> > > @@ -3089,6 +3101,17 @@ static void virtio_net_get_features(VirtIODevice *vdev, uint64_t *features,
> > >       virtio_add_feature_ex(features, VIRTIO_NET_F_MAC);
> > > +    if (n->peers_wait_incoming) {
> > > +        /*
> > > +         * Excessive feature set is OK for early initialization when
> > > +         * we wait for local incoming migration: actual guest-negotiated
> > > +         * features will come with migration stream anyway. And we are sure
> > > +         * that we support same host-features as source, because the backend
> > > +         * is the same (the same TAP device, for example).
> > > +         */
> > > +        return;
> > > +    }
> > > +
> > >       if (!peer_has_vnet_hdr(n)) {
> > >           virtio_clear_feature_ex(features, VIRTIO_NET_F_CSUM);
> > >           virtio_clear_feature_ex(features, VIRTIO_NET_F_HOST_TSO4);
> > > @@ -3179,6 +3202,18 @@ static void virtio_net_get_features(VirtIODevice *vdev, uint64_t *features,
> > >       }
> > >   }
> > > +static bool virtio_net_update_host_features(VirtIONet *n, Error **errp)
> > > +{
> > > +    ERRP_GUARD();
> > > +    VirtIODevice *vdev = VIRTIO_DEVICE(n);
> > > +
> > > +    peer_test_vnet_hdr(n);
> > > +
> > > +    virtio_net_get_features(vdev, &vdev->host_features, errp);
> > > +
> > > +    return !*errp;
> > > +}
> > > +
> > >   static int virtio_net_post_load_device(void *opaque, int version_id)
> > >   {
> > >       VirtIONet *n = opaque;
> > > @@ -3300,6 +3335,9 @@ struct VirtIONetMigTmp {
> > >       uint16_t        curr_queue_pairs_1;
> > >       uint8_t         has_ufo;
> > >       uint32_t        has_vnet_hdr;
> > > +
> > > +    NetClientState *ncs;
> > > +    uint32_t max_queue_pairs;
> > >   };
> > >   /* The 2nd and subsequent tx_waiting flags are loaded later than
> > > @@ -3569,6 +3607,57 @@ static const VMStateDescription vhost_user_net_backend_state = {
> > >       }
> > >   };
> > > +static bool virtio_net_migrate_local(void *opaque, int version_id)
> > > +{
> > > +    VirtIONet *n = opaque;
> > > +
> > > +    return migrate_local() && n->local_migration;
> > > +}
> > > +
> > > +static int virtio_net_nic_pre_save(void *opaque)
> > > +{
> > > +    struct VirtIONetMigTmp *tmp = opaque;
> > > +
> > > +    tmp->ncs = tmp->parent->nic->ncs;
> > > +    tmp->max_queue_pairs = tmp->parent->max_queue_pairs;
> > > +
> > > +    return 0;
> > > +}
> > > +
> > > +static int virtio_net_nic_pre_load(void *opaque)
> > > +{
> > > +    /* Reuse the pointer setup from save */
> > > +    virtio_net_nic_pre_save(opaque);
> > > +
> > > +    return 0;
> > > +}
> > > +
> > > +static int virtio_net_nic_post_load(void *opaque, int version_id)
> > > +{
> > > +    struct VirtIONetMigTmp *tmp = opaque;
> > > +    Error *local_err = NULL;
> > > +
> > > +    if (!virtio_net_update_host_features(tmp->parent, &local_err)) {
> > > +        error_report_err(local_err);
> > > +        return -EINVAL;
> > > +    }
> > > +
> > > +    return 0;
> > > +}
> > > +
> > > +static const VMStateDescription vmstate_virtio_net_nic = {
> > > +    .name      = "virtio-net-nic",
> > > +    .pre_load  = virtio_net_nic_pre_load,
> > > +    .pre_save  = virtio_net_nic_pre_save,
> > > +    .post_load  = virtio_net_nic_post_load,
> > > +    .fields    = (const VMStateField[]) {
> > > +        VMSTATE_VARRAY_UINT32(ncs, struct VirtIONetMigTmp,
> > > +                              max_queue_pairs, 0, vmstate_net_peer_backend,
> > > +                              NetClientState),
> > > +        VMSTATE_END_OF_LIST()
> > > +    },
> > > +};
> > > +
> > >   static const VMStateDescription vmstate_virtio_net_device = {
> > >       .name = "virtio-net-device",
> > >       .version_id = VIRTIO_NET_VM_VERSION,
> > > @@ -3600,6 +3689,9 @@ static const VMStateDescription vmstate_virtio_net_device = {
> > >            * but based on the uint.
> > >            */
> > >           VMSTATE_BUFFER_POINTER_UNSAFE(vlans, VirtIONet, 0, MAX_VLAN >> 3),
> > > +        VMSTATE_WITH_TMP_TEST(VirtIONet, virtio_net_migrate_local,
> > > +                              struct VirtIONetMigTmp,
> > > +                              vmstate_virtio_net_nic),
> > >           VMSTATE_WITH_TMP(VirtIONet, struct VirtIONetMigTmp,
> > >                            vmstate_virtio_net_has_vnet),
> > >           VMSTATE_UINT8(mac_table.multi_overflow, VirtIONet),
> > > @@ -3864,6 +3956,42 @@ static bool failover_hide_primary_device(DeviceListener *listener,
> > >       return qatomic_read(&n->failover_primary_hidden);
> > >   }
> > > +static bool virtio_net_check_peers_wait_incoming(VirtIONet *n, bool *waiting,
> > > +                                                 Error **errp)
> > > +{
> > > +    bool has_waiting = false;
> > > +    bool has_not_waiting = false;
> > > +
> > > +    for (int i = 0; i < n->max_queue_pairs; i++) {
> > > +        NetClientState *peer = n->nic->ncs[i].peer;
> > > +        if (!peer) {
> > > +            continue;
> > > +        }
> > > +
> > > +        if (peer->info->is_wait_incoming &&
> > > +            peer->info->is_wait_incoming(peer)) {
> > > +            has_waiting = true;
> > > +        } else {
> > > +            has_not_waiting = true;
> > > +        }
> > > +
> > > +        if (has_waiting && has_not_waiting) {
> > > +            error_setg(errp, "Mixed peer states: some peers wait for incoming "
> > > +                       "migration while others don't");
> > > +            return false;
> > > +        }
> > > +    }
> > > +
> > > +    if (has_waiting && !runstate_check(RUN_STATE_INMIGRATE)) {
> > > +        error_setg(errp, "Peers wait for incoming, but it's not an incoming "
> > > +                   "migration.");
> > > +        return false;
> > > +    }
> > > +
> > > +    *waiting = has_waiting;
> > > +    return true;
> > > +}
> > > +
> > >   static void virtio_net_device_realize(DeviceState *dev, Error **errp)
> > >   {
> > >       VirtIODevice *vdev = VIRTIO_DEVICE(dev);
> > > @@ -4001,6 +4129,12 @@ static void virtio_net_device_realize(DeviceState *dev, Error **errp)
> > >           n->nic->ncs[i].do_not_pad = true;
> > >       }
> > > +    if (!virtio_net_check_peers_wait_incoming(n, &n->peers_wait_incoming,
> > > +                                              errp)) {
> > > +        virtio_cleanup(vdev);
> > > +        return;
> > > +    }
> > > +
> > >       peer_test_vnet_hdr(n);
> > >       if (peer_has_vnet_hdr(n)) {
> > >           n->host_hdr_len = sizeof(struct virtio_net_hdr);
> > > @@ -4310,6 +4444,7 @@ static const Property virtio_net_properties[] = {
> > >                                  host_features_ex,
> > >                                  VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO_CSUM,
> > >                                  true),
> > > +    DEFINE_PROP_BOOL("local-migration", VirtIONet, local_migration, true),
> > >   };
> > >   static void virtio_net_class_init(ObjectClass *klass, const void *data)
> > > diff --git a/include/hw/virtio/virtio-net.h b/include/hw/virtio/virtio-net.h
> > > index 371e3764282..0c14e314409 100644
> > > --- a/include/hw/virtio/virtio-net.h
> > > +++ b/include/hw/virtio/virtio-net.h
> > > @@ -230,6 +230,8 @@ struct VirtIONet {
> > >       struct EBPFRSSContext ebpf_rss;
> > >       uint32_t nr_ebpf_rss_fds;
> > >       char **ebpf_rss_fds;
> > > +    bool peers_wait_incoming;
> > > +    bool local_migration;
> > >   };
> > >   size_t virtio_net_handle_ctrl_iov(VirtIODevice *vdev,
> > > diff --git a/include/net/net.h b/include/net/net.h
> > > index aa34043b1ac..d4cf399d4a8 100644
> > > --- a/include/net/net.h
> > > +++ b/include/net/net.h
> > > @@ -82,6 +82,7 @@ typedef void (SocketReadStateFinalize)(SocketReadState *rs);
> > >   typedef void (NetAnnounce)(NetClientState *);
> > >   typedef bool (SetSteeringEBPF)(NetClientState *, int);
> > >   typedef bool (NetCheckPeerType)(NetClientState *, ObjectClass *, Error **);
> > > +typedef bool (IsWaitIncoming)(NetClientState *);
> > >   typedef struct vhost_net *(GetVHostNet)(NetClientState *nc);
> > >   typedef struct NetClientInfo {
> > > @@ -110,6 +111,7 @@ typedef struct NetClientInfo {
> > >       NetAnnounce *announce;
> > >       SetSteeringEBPF *set_steering_ebpf;
> > >       NetCheckPeerType *check_peer_type;
> > > +    IsWaitIncoming *is_wait_incoming;
> > >       GetVHostNet *get_vhost_net;
> > >       const VMStateDescription *backend_vmsd;
> > >   } NetClientInfo;
> > > -- 
> > > 2.52.0
> > 
> 
> 
> -- 
> Best regards,
> Vladimir
> 

-- 
Peter Xu