* [PATCH v16 0/8] virtio-net: live-TAP local migration
@ 2026-05-22 12:05 Vladimir Sementsov-Ogievskiy
2026-05-22 12:05 ` [PATCH v16 1/8] net/tap: move vhost-net open() calls to tap_parse_vhost_fds() Vladimir Sementsov-Ogievskiy
` (8 more replies)
0 siblings, 9 replies; 25+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2026-05-22 12:05 UTC (permalink / raw)
To: jasowang, mst
Cc: armbru, peterx, farosas, raphael.s.norwitz, bchaney, vsementsov,
qemu-devel, berrange, pbonzini, yc-core
Hi all!
Here is a new migration parameter "local", which allows to
enable local migration of TAP virtio-net backend (and maybe other
devices and backends in future), including its properties and open
fds.
With this new option, management software doesn't need to initialize
new TAP and do a switch to it. Nothing should be done around
virtio-net in local migration: it just migrates and continues to use
same TAP device. So we avoid extra logic in management software, extra
allocations in kernel (for new TAP), and corresponding extra delay in
migration downtime.
v16: rebase on master (on top of "[PATCH v4 00/13] net: refactoring and fixes"),
01-08: add r-b by Ben
06: add a-b by Markus
07: small fix, exclude vnet_hdr= and ifname= arguments when
incoming-fds is set (they are not allowed actually, and
netdev_add fails), keep r-b by Ben.
Based-on: <20260318113144.15697-1-vsementsov@yandex-team.ru>
"[PATCH v4 00/13] net: refactoring and fixes"
Vladimir Sementsov-Ogievskiy (8):
net/tap: move vhost-net open() calls to tap_parse_vhost_fds()
net/tap: move vhost initialization to tap_setup_vhost()
qapi: add local migration parameter
net: introduce vmstate_net_peer_backend
virtio-net: support local migration of backend
net/tap: support local migration with virtio-net
tests/functional: add skipWithoutSudo() decorator
tests/functional: add test_tap_migration
hw/core/machine.c | 1 +
hw/i386/pc_q35.c | 1 +
hw/net/virtio-net.c | 137 +++++-
include/hw/virtio/virtio-net.h | 2 +
include/migration/misc.h | 2 +
include/net/net.h | 6 +
migration/options.c | 18 +-
net/net.c | 47 ++
net/tap.c | 246 ++++++++--
qapi/migration.json | 12 +-
qapi/net.json | 10 +-
tests/functional/qemu_test/decorators.py | 16 +
tests/functional/x86_64/meson.build | 1 +
tests/functional/x86_64/test_tap_migration.py | 458 ++++++++++++++++++
14 files changed, 906 insertions(+), 51 deletions(-)
create mode 100755 tests/functional/x86_64/test_tap_migration.py
--
2.52.0
^ permalink raw reply [flat|nested] 25+ messages in thread
* [PATCH v16 1/8] net/tap: move vhost-net open() calls to tap_parse_vhost_fds()
2026-05-22 12:05 [PATCH v16 0/8] virtio-net: live-TAP local migration Vladimir Sementsov-Ogievskiy
@ 2026-05-22 12:05 ` Vladimir Sementsov-Ogievskiy
2026-05-22 12:05 ` [PATCH v16 2/8] net/tap: move vhost initialization to tap_setup_vhost() Vladimir Sementsov-Ogievskiy
` (7 subsequent siblings)
8 siblings, 0 replies; 25+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2026-05-22 12:05 UTC (permalink / raw)
To: jasowang, mst
Cc: armbru, peterx, farosas, raphael.s.norwitz, bchaney, vsementsov,
qemu-devel, berrange, pbonzini, yc-core
1. Simplify code path: get vhostfds for all cases in one function.
2. Prepare for further tap-fd-migraton feature, when we'll need to
postpone vhost initialization up to post-load stage.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Reviewed-by: Ben Chaney <bchaney@akamai.com>
---
net/tap.c | 39 ++++++++++++++++++++++-----------------
1 file changed, 22 insertions(+), 17 deletions(-)
diff --git a/net/tap.c b/net/tap.c
index 57ffb09885c..d941c67895e 100644
--- a/net/tap.c
+++ b/net/tap.c
@@ -736,8 +736,7 @@ static bool net_init_tap_one(const NetdevTapOptions *tap, NetClientState *peer,
}
}
- if (tap->has_vhost ? tap->vhost :
- (vhostfd != -1) || (tap->has_vhostforce && tap->vhostforce)) {
+ if (vhostfd != -1) {
VhostNetOptions options;
options.backend_type = VHOST_BACKEND_TYPE_KERNEL;
@@ -747,17 +746,6 @@ static bool net_init_tap_one(const NetdevTapOptions *tap, NetClientState *peer,
} else {
options.busyloop_timeout = 0;
}
-
- if (vhostfd == -1) {
- vhostfd = open("/dev/vhost-net", O_RDWR);
- if (vhostfd < 0) {
- error_setg_file_open(errp, errno, "/dev/vhost-net");
- goto failed;
- }
- if (!qemu_set_blocking(vhostfd, false, errp)) {
- goto failed;
- }
- }
options.opaque = (void *)(uintptr_t)vhostfd;
options.nvqs = 2;
options.feature_bits = kernel_feature_bits;
@@ -843,14 +831,31 @@ static int tap_parse_fds_and_queues(const NetdevTapOptions *tap, int **fds,
static bool tap_parse_vhost_fds(const NetdevTapOptions *tap, int **vhost_fds,
int queues, Error **errp)
{
- if (!(tap->vhostfd || tap->vhostfds)) {
+ bool need_vhost = tap->has_vhost ? tap->vhost :
+ ((tap->vhostfd || tap->vhostfds) ||
+ (tap->has_vhostforce && tap->vhostforce));
+
+ if (!need_vhost) {
*vhost_fds = NULL;
return true;
}
- if (net_parse_fds(tap->vhostfd ?: tap->vhostfds,
- vhost_fds, queues, errp) < 0) {
- return false;
+ if (tap->vhostfd || tap->vhostfds) {
+ if (net_parse_fds(tap->vhostfd ?: tap->vhostfds,
+ vhost_fds, queues, errp) < 0) {
+ return false;
+ }
+ } else {
+ *vhost_fds = g_new(int, queues);
+ for (int i = 0; i < queues; i++) {
+ int vhostfd = open("/dev/vhost-net", O_RDWR);
+ if (vhostfd < 0) {
+ error_setg_file_open(errp, errno, "/dev/vhost-net");
+ net_free_fds(*vhost_fds, i);
+ return false;
+ }
+ (*vhost_fds)[i] = vhostfd;
+ }
}
if (!unblock_fds(*vhost_fds, queues, errp)) {
--
2.52.0
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [PATCH v16 2/8] net/tap: move vhost initialization to tap_setup_vhost()
2026-05-22 12:05 [PATCH v16 0/8] virtio-net: live-TAP local migration Vladimir Sementsov-Ogievskiy
2026-05-22 12:05 ` [PATCH v16 1/8] net/tap: move vhost-net open() calls to tap_parse_vhost_fds() Vladimir Sementsov-Ogievskiy
@ 2026-05-22 12:05 ` Vladimir Sementsov-Ogievskiy
2026-05-22 12:05 ` [PATCH v16 3/8] qapi: add local migration parameter Vladimir Sementsov-Ogievskiy
` (6 subsequent siblings)
8 siblings, 0 replies; 25+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2026-05-22 12:05 UTC (permalink / raw)
To: jasowang, mst
Cc: armbru, peterx, farosas, raphael.s.norwitz, bchaney, vsementsov,
qemu-devel, berrange, pbonzini, yc-core
Make a new helper function in a way it can be reused later for
TAP fd-migration feature: we'll need to initialize vhost in a later
point when we doesn't have access to QAPI parameters.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Reviewed-by: Ben Chaney <bchaney@akamai.com>
---
net/tap.c | 62 ++++++++++++++++++++++++++++++++++---------------------
1 file changed, 38 insertions(+), 24 deletions(-)
diff --git a/net/tap.c b/net/tap.c
index d941c67895e..9d6213fc3e5 100644
--- a/net/tap.c
+++ b/net/tap.c
@@ -71,6 +71,8 @@ static const int kernel_feature_bits[] = {
typedef struct TAPState {
NetClientState nc;
int fd;
+ int vhostfd;
+ uint32_t vhost_busyloop_timeout;
char down_script[1024];
char down_script_arg[128];
uint8_t buf[NET_BUFSIZE];
@@ -702,6 +704,38 @@ static int net_tap_init(const NetdevTapOptions *tap, int *vnet_hdr,
return fd;
}
+static bool tap_setup_vhost(TAPState *s, Error **errp)
+{
+ VhostNetOptions options;
+
+ if (s->vhostfd == -1) {
+ return true;
+ }
+
+ options.backend_type = VHOST_BACKEND_TYPE_KERNEL;
+ options.net_backend = &s->nc;
+ options.busyloop_timeout = s->vhost_busyloop_timeout;
+ options.opaque = (void *)(uintptr_t)s->vhostfd;
+ options.nvqs = 2;
+ options.feature_bits = kernel_feature_bits;
+ options.get_acked_features = NULL;
+ options.save_acked_features = NULL;
+ options.max_tx_queue_size = 0;
+ options.is_vhost_user = false;
+
+ s->vhost_net = vhost_net_init(&options);
+ if (!s->vhost_net) {
+ error_setg(errp,
+ "vhost-net requested but could not be initialized");
+ return false;
+ }
+
+ /* vhostfd ownership is passed to s->vhost_net */
+ s->vhostfd = -1;
+
+ return true;
+}
+
static bool net_init_tap_one(const NetdevTapOptions *tap, NetClientState *peer,
const char *name,
const char *ifname, const char *script,
@@ -736,30 +770,10 @@ static bool net_init_tap_one(const NetdevTapOptions *tap, NetClientState *peer,
}
}
- if (vhostfd != -1) {
- VhostNetOptions options;
-
- options.backend_type = VHOST_BACKEND_TYPE_KERNEL;
- options.net_backend = &s->nc;
- if (tap->has_poll_us) {
- options.busyloop_timeout = tap->poll_us;
- } else {
- options.busyloop_timeout = 0;
- }
- options.opaque = (void *)(uintptr_t)vhostfd;
- options.nvqs = 2;
- options.feature_bits = kernel_feature_bits;
- options.get_acked_features = NULL;
- options.save_acked_features = NULL;
- options.max_tx_queue_size = 0;
- options.is_vhost_user = false;
-
- s->vhost_net = vhost_net_init(&options);
- if (!s->vhost_net) {
- error_setg(errp,
- "vhost-net requested but could not be initialized");
- goto failed;
- }
+ s->vhostfd = vhostfd;
+ s->vhost_busyloop_timeout = tap->has_poll_us ? tap->poll_us : 0;
+ if (!tap_setup_vhost(s, errp)) {
+ return false;
}
return true;
--
2.52.0
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [PATCH v16 3/8] qapi: add local migration parameter
2026-05-22 12:05 [PATCH v16 0/8] virtio-net: live-TAP local migration Vladimir Sementsov-Ogievskiy
2026-05-22 12:05 ` [PATCH v16 1/8] net/tap: move vhost-net open() calls to tap_parse_vhost_fds() Vladimir Sementsov-Ogievskiy
2026-05-22 12:05 ` [PATCH v16 2/8] net/tap: move vhost initialization to tap_setup_vhost() Vladimir Sementsov-Ogievskiy
@ 2026-05-22 12:05 ` Vladimir Sementsov-Ogievskiy
2026-05-22 12:05 ` [PATCH v16 4/8] net: introduce vmstate_net_peer_backend Vladimir Sementsov-Ogievskiy
` (5 subsequent siblings)
8 siblings, 0 replies; 25+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2026-05-22 12:05 UTC (permalink / raw)
To: jasowang, mst
Cc: armbru, peterx, farosas, raphael.s.norwitz, bchaney, vsementsov,
qemu-devel, berrange, pbonzini, yc-core, Eric Blake
We are going to implement local-migration feature: some devices will be
able to transfer open file descriptors through migration stream (which
must UNIX domain socket for that purpose). This allows to transfer the
whole backend state without reconnecting and restarting the backend
service. For example, virtio-net will migrate its attached TAP netdev,
together with its connected file descriptors.
In this commit we introduce a migration parameter, which enables
the feature for devices that support it (none at the moment).
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Acked-by: Markus Armbruster <armbru@redhat.com>
Acked-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Ben Chaney <bchaney@akamai.com>
---
include/migration/misc.h | 2 ++
migration/options.c | 18 +++++++++++++++++-
qapi/migration.json | 12 ++++++++++--
3 files changed, 29 insertions(+), 3 deletions(-)
diff --git a/include/migration/misc.h b/include/migration/misc.h
index 3159a5e53c3..b14dc70ea3d 100644
--- a/include/migration/misc.h
+++ b/include/migration/misc.h
@@ -156,4 +156,6 @@ bool multifd_device_state_save_thread_should_exit(void);
void multifd_abort_device_state_save_threads(void);
bool multifd_join_device_state_save_threads(void);
+bool migrate_local(void);
+
#endif
diff --git a/migration/options.c b/migration/options.c
index 5cbfd29099b..ba5f9440a08 100644
--- a/migration/options.c
+++ b/migration/options.c
@@ -14,6 +14,7 @@
#include "qemu/osdep.h"
#include "qemu/error-report.h"
#include "qemu/units.h"
+#include "qapi/util.h"
#include "exec/target_page.h"
#include "qapi/clone-visitor.h"
#include "qapi/error.h"
@@ -25,6 +26,7 @@
#include "migration/colo.h"
#include "migration/cpr.h"
#include "migration/misc.h"
+#include "migration/options.h"
#include "migration.h"
#include "migration-stats.h"
#include "qemu-file.h"
@@ -348,6 +350,12 @@ bool migrate_mapped_ram(void)
return s->capabilities[MIGRATION_CAPABILITY_MAPPED_RAM];
}
+bool migrate_local(void)
+{
+ MigrationState *s = migrate_get_current();
+ return s->parameters.local;
+}
+
bool migrate_ignore_shared(void)
{
MigrationState *s = migrate_get_current();
@@ -1076,7 +1084,7 @@ static void migrate_mark_all_params_present(MigrationParameters *p)
&p->has_announce_step, &p->has_block_bitmap_mapping,
&p->has_x_vcpu_dirty_limit_period, &p->has_vcpu_dirty_limit,
&p->has_mode, &p->has_zero_page_detection, &p->has_direct_io,
- &p->has_x_rdma_chunk_size, &p->has_cpr_exec_command,
+ &p->has_x_rdma_chunk_size, &p->has_cpr_exec_command, &p->has_local,
};
len = ARRAY_SIZE(has_fields);
@@ -1424,6 +1432,10 @@ static void migrate_params_test_apply(MigrationParameters *params,
qapi_free_strList(dest->cpr_exec_command);
dest->cpr_exec_command = QAPI_CLONE(strList, params->cpr_exec_command);
}
+
+ if (params->has_local) {
+ dest->local = params->local;
+ }
}
static void migrate_params_apply(MigrationParameters *params)
@@ -1556,6 +1568,10 @@ static void migrate_params_apply(MigrationParameters *params)
s->parameters.cpr_exec_command =
QAPI_CLONE(strList, params->cpr_exec_command);
}
+
+ if (params->has_local) {
+ s->parameters.local = params->local;
+ }
}
void qmp_migrate_set_parameters(MigrationParameters *params, Error **errp)
diff --git a/qapi/migration.json b/qapi/migration.json
index 27a79705569..bc331815da0 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -836,7 +836,8 @@
'zero-page-detection',
'direct-io',
{ 'name': 'x-rdma-chunk-size', 'features': [ 'unstable' ] },
- 'cpr-exec-command'] }
+ 'cpr-exec-command',
+ 'local'] }
##
# @migrate-set-parameters:
@@ -1018,6 +1019,12 @@
# Must be set to the same value on both source and destination
# before migration starts. (Since 11.1)
#
+# @local: Enable local migration for devices that support it. Backend
+# state and its file descriptors can then be passed to the
+# destination in the migration channel. The migration channel
+# must be a Unix domain socket. Usually needs to be enabled per
+# device. (Since 11.1)
+#
# Features:
#
# @unstable: Members @x-checkpoint-delay, @x-rdma-chunk-size, and
@@ -1059,7 +1066,8 @@
'*direct-io': 'bool',
'*x-rdma-chunk-size': { 'type': 'uint64',
'features': [ 'unstable' ] },
- '*cpr-exec-command': [ 'str' ]} }
+ '*cpr-exec-command': [ 'str' ],
+ '*local': 'bool' } }
##
# @query-migrate-parameters:
--
2.52.0
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [PATCH v16 4/8] net: introduce vmstate_net_peer_backend
2026-05-22 12:05 [PATCH v16 0/8] virtio-net: live-TAP local migration Vladimir Sementsov-Ogievskiy
` (2 preceding siblings ...)
2026-05-22 12:05 ` [PATCH v16 3/8] qapi: add local migration parameter Vladimir Sementsov-Ogievskiy
@ 2026-05-22 12:05 ` Vladimir Sementsov-Ogievskiy
2026-05-22 12:05 ` [PATCH v16 5/8] virtio-net: support local migration of backend Vladimir Sementsov-Ogievskiy
` (4 subsequent siblings)
8 siblings, 0 replies; 25+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2026-05-22 12:05 UTC (permalink / raw)
To: jasowang, mst
Cc: armbru, peterx, farosas, raphael.s.norwitz, bchaney, vsementsov,
qemu-devel, berrange, pbonzini, yc-core
To implement backend migration in virtio-net in the next commit, we need
a generic API to migrate net backend. Here is it.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Reviewed-by: Ben Chaney <bchaney@akamai.com>
---
include/net/net.h | 4 ++++
net/net.c | 47 +++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 51 insertions(+)
diff --git a/include/net/net.h b/include/net/net.h
index 45bc86fc86b..aa34043b1ac 100644
--- a/include/net/net.h
+++ b/include/net/net.h
@@ -5,6 +5,7 @@
#include "qapi/qapi-types-net.h"
#include "net/queue.h"
#include "hw/core/qdev-properties-system.h"
+#include "migration/vmstate.h"
#define MAC_FMT "%02X:%02X:%02X:%02X:%02X:%02X"
#define MAC_ARG(x) ((uint8_t *)(x))[0], ((uint8_t *)(x))[1], \
@@ -110,6 +111,7 @@ typedef struct NetClientInfo {
SetSteeringEBPF *set_steering_ebpf;
NetCheckPeerType *check_peer_type;
GetVHostNet *get_vhost_net;
+ const VMStateDescription *backend_vmsd;
} NetClientInfo;
struct NetClientState {
@@ -354,4 +356,6 @@ static inline bool net_peer_needs_padding(NetClientState *nc)
return nc->peer && !nc->peer->do_not_pad;
}
+extern const VMStateInfo vmstate_net_peer_backend;
+
#endif
diff --git a/net/net.c b/net/net.c
index 2892f1730d1..8bccb1880a1 100644
--- a/net/net.c
+++ b/net/net.c
@@ -58,6 +58,7 @@
#include "qapi/string-output-visitor.h"
#include "qapi/qobject-input-visitor.h"
#include "standard-headers/linux/virtio_net.h"
+#include "migration/vmstate.h"
/* Net bridge is currently not supported for W32. */
#if !defined(_WIN32)
@@ -2173,3 +2174,49 @@ int net_fill_rstate(SocketReadState *rs, const uint8_t *buf, int size)
assert(size == 0);
return 0;
}
+
+static int get_peer_backend(QEMUFile *f, void *pv, size_t size,
+ const VMStateField *field)
+{
+ NetClientState *nc = pv;
+ Error *local_err = NULL;
+ int ret;
+
+ if (!nc->peer) {
+ return -EINVAL;
+ }
+ nc = nc->peer;
+
+ ret = vmstate_load_state(f, nc->info->backend_vmsd, nc, 0, &local_err);
+ if (ret < 0) {
+ error_report_err(local_err);
+ }
+
+ return ret;
+}
+
+static int put_peer_backend(QEMUFile *f, void *pv, size_t size,
+ const VMStateField *field, JSONWriter *vmdesc)
+{
+ NetClientState *nc = pv;
+ Error *local_err = NULL;
+ int ret;
+
+ if (!nc->peer) {
+ return -EINVAL;
+ }
+ nc = nc->peer;
+
+ ret = vmstate_save_state(f, nc->info->backend_vmsd, nc, 0, &local_err);
+ if (ret < 0) {
+ error_report_err(local_err);
+ }
+
+ return ret;
+}
+
+const VMStateInfo vmstate_net_peer_backend = {
+ .name = "virtio-net-nic-nc-backend",
+ .get = get_peer_backend,
+ .put = put_peer_backend,
+};
--
2.52.0
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [PATCH v16 5/8] virtio-net: support local migration of backend
2026-05-22 12:05 [PATCH v16 0/8] virtio-net: live-TAP local migration Vladimir Sementsov-Ogievskiy
` (3 preceding siblings ...)
2026-05-22 12:05 ` [PATCH v16 4/8] net: introduce vmstate_net_peer_backend Vladimir Sementsov-Ogievskiy
@ 2026-05-22 12:05 ` Vladimir Sementsov-Ogievskiy
2026-05-24 9:09 ` Michael S. Tsirkin
2026-05-22 12:05 ` [PATCH v16 6/8] net/tap: support local migration with virtio-net Vladimir Sementsov-Ogievskiy
` (3 subsequent siblings)
8 siblings, 1 reply; 25+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2026-05-22 12:05 UTC (permalink / raw)
To: jasowang, mst
Cc: armbru, peterx, farosas, raphael.s.norwitz, bchaney, vsementsov,
qemu-devel, berrange, pbonzini, yc-core,
Philippe Mathieu-Daudé, Zhao Liu, Richard Henderson
Add virtio-net option local-migration, which is true by default,
but false for older machine types, which doesn't support the feature.
When both global migration parameter "local" and new virtio-net
parameter "local-migration" are true, virtio-net transfer the whole
net backend to the destination, including open file descriptors.
Of-course, its only for local migration and the channel must be
UNIX domain socket.
This way management tool should not care about creating new TAP, and
should not handle switching to it. Migration downtime become shorter.
Support for TAP will come in the next commit.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Reviewed-by: Ben Chaney <bchaney@akamai.com>
---
hw/core/machine.c | 1 +
hw/i386/pc_q35.c | 1 +
hw/net/virtio-net.c | 137 ++++++++++++++++++++++++++++++++-
include/hw/virtio/virtio-net.h | 2 +
include/net/net.h | 2 +
5 files changed, 142 insertions(+), 1 deletion(-)
diff --git a/hw/core/machine.c b/hw/core/machine.c
index 63baff859f3..619e80c1cb3 100644
--- a/hw/core/machine.c
+++ b/hw/core/machine.c
@@ -41,6 +41,7 @@
GlobalProperty hw_compat_11_0[] = {
{ "chardev-vc", "encoding", "cp437" },
+ { TYPE_VIRTIO_NET, "local-migration", "false" },
};
const size_t hw_compat_11_0_len = G_N_ELEMENTS(hw_compat_11_0);
diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
index d8fed698c72..b5c0e302d59 100644
--- a/hw/i386/pc_q35.c
+++ b/hw/i386/pc_q35.c
@@ -368,6 +368,7 @@ static void pc_q35_machine_options(MachineClass *m)
static void pc_q35_machine_11_1_options(MachineClass *m)
{
pc_q35_machine_options(m);
+ compat_props_add(m->compat_props, hw_compat_11_0, hw_compat_11_0_len);
}
DEFINE_Q35_MACHINE_AS_LATEST(11, 1);
diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index 2a5d642a647..158b9247a58 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -38,8 +38,10 @@
#include "qapi/qapi-events-migration.h"
#include "hw/virtio/virtio-access.h"
#include "migration/misc.h"
+#include "migration/options.h"
#include "standard-headers/linux/ethtool.h"
#include "system/system.h"
+#include "system/runstate.h"
#include "system/replay.h"
#include "trace.h"
#include "monitor/qdev.h"
@@ -3060,7 +3062,17 @@ static void virtio_net_set_multiqueue(VirtIONet *n, int multiqueue)
n->multiqueue = multiqueue;
virtio_net_change_num_queues(n, max * 2 + 1);
- virtio_net_set_queue_pairs(n);
+ /*
+ * virtio_net_set_multiqueue() called from set_features(0) on early
+ * reset, when peer may wait for incoming (and is not initialized
+ * yet).
+ * Don't worry about it: virtio_net_set_queue_pairs() will be called
+ * later form virtio_net_post_load_device(), and anyway will be
+ * noop for local incoming migration with live backend passing.
+ */
+ if (!n->peers_wait_incoming) {
+ virtio_net_set_queue_pairs(n);
+ }
}
static int virtio_net_pre_load_queues(VirtIODevice *vdev, uint32_t n)
@@ -3089,6 +3101,17 @@ static void virtio_net_get_features(VirtIODevice *vdev, uint64_t *features,
virtio_add_feature_ex(features, VIRTIO_NET_F_MAC);
+ if (n->peers_wait_incoming) {
+ /*
+ * Excessive feature set is OK for early initialization when
+ * we wait for local incoming migration: actual guest-negotiated
+ * features will come with migration stream anyway. And we are sure
+ * that we support same host-features as source, because the backend
+ * is the same (the same TAP device, for example).
+ */
+ return;
+ }
+
if (!peer_has_vnet_hdr(n)) {
virtio_clear_feature_ex(features, VIRTIO_NET_F_CSUM);
virtio_clear_feature_ex(features, VIRTIO_NET_F_HOST_TSO4);
@@ -3179,6 +3202,18 @@ static void virtio_net_get_features(VirtIODevice *vdev, uint64_t *features,
}
}
+static bool virtio_net_update_host_features(VirtIONet *n, Error **errp)
+{
+ ERRP_GUARD();
+ VirtIODevice *vdev = VIRTIO_DEVICE(n);
+
+ peer_test_vnet_hdr(n);
+
+ virtio_net_get_features(vdev, &vdev->host_features, errp);
+
+ return !*errp;
+}
+
static int virtio_net_post_load_device(void *opaque, int version_id)
{
VirtIONet *n = opaque;
@@ -3300,6 +3335,9 @@ struct VirtIONetMigTmp {
uint16_t curr_queue_pairs_1;
uint8_t has_ufo;
uint32_t has_vnet_hdr;
+
+ NetClientState *ncs;
+ uint32_t max_queue_pairs;
};
/* The 2nd and subsequent tx_waiting flags are loaded later than
@@ -3569,6 +3607,57 @@ static const VMStateDescription vhost_user_net_backend_state = {
}
};
+static bool virtio_net_migrate_local(void *opaque, int version_id)
+{
+ VirtIONet *n = opaque;
+
+ return migrate_local() && n->local_migration;
+}
+
+static int virtio_net_nic_pre_save(void *opaque)
+{
+ struct VirtIONetMigTmp *tmp = opaque;
+
+ tmp->ncs = tmp->parent->nic->ncs;
+ tmp->max_queue_pairs = tmp->parent->max_queue_pairs;
+
+ return 0;
+}
+
+static int virtio_net_nic_pre_load(void *opaque)
+{
+ /* Reuse the pointer setup from save */
+ virtio_net_nic_pre_save(opaque);
+
+ return 0;
+}
+
+static int virtio_net_nic_post_load(void *opaque, int version_id)
+{
+ struct VirtIONetMigTmp *tmp = opaque;
+ Error *local_err = NULL;
+
+ if (!virtio_net_update_host_features(tmp->parent, &local_err)) {
+ error_report_err(local_err);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static const VMStateDescription vmstate_virtio_net_nic = {
+ .name = "virtio-net-nic",
+ .pre_load = virtio_net_nic_pre_load,
+ .pre_save = virtio_net_nic_pre_save,
+ .post_load = virtio_net_nic_post_load,
+ .fields = (const VMStateField[]) {
+ VMSTATE_VARRAY_UINT32(ncs, struct VirtIONetMigTmp,
+ max_queue_pairs, 0, vmstate_net_peer_backend,
+ NetClientState),
+ VMSTATE_END_OF_LIST()
+ },
+};
+
static const VMStateDescription vmstate_virtio_net_device = {
.name = "virtio-net-device",
.version_id = VIRTIO_NET_VM_VERSION,
@@ -3600,6 +3689,9 @@ static const VMStateDescription vmstate_virtio_net_device = {
* but based on the uint.
*/
VMSTATE_BUFFER_POINTER_UNSAFE(vlans, VirtIONet, 0, MAX_VLAN >> 3),
+ VMSTATE_WITH_TMP_TEST(VirtIONet, virtio_net_migrate_local,
+ struct VirtIONetMigTmp,
+ vmstate_virtio_net_nic),
VMSTATE_WITH_TMP(VirtIONet, struct VirtIONetMigTmp,
vmstate_virtio_net_has_vnet),
VMSTATE_UINT8(mac_table.multi_overflow, VirtIONet),
@@ -3864,6 +3956,42 @@ static bool failover_hide_primary_device(DeviceListener *listener,
return qatomic_read(&n->failover_primary_hidden);
}
+static bool virtio_net_check_peers_wait_incoming(VirtIONet *n, bool *waiting,
+ Error **errp)
+{
+ bool has_waiting = false;
+ bool has_not_waiting = false;
+
+ for (int i = 0; i < n->max_queue_pairs; i++) {
+ NetClientState *peer = n->nic->ncs[i].peer;
+ if (!peer) {
+ continue;
+ }
+
+ if (peer->info->is_wait_incoming &&
+ peer->info->is_wait_incoming(peer)) {
+ has_waiting = true;
+ } else {
+ has_not_waiting = true;
+ }
+
+ if (has_waiting && has_not_waiting) {
+ error_setg(errp, "Mixed peer states: some peers wait for incoming "
+ "migration while others don't");
+ return false;
+ }
+ }
+
+ if (has_waiting && !runstate_check(RUN_STATE_INMIGRATE)) {
+ error_setg(errp, "Peers wait for incoming, but it's not an incoming "
+ "migration.");
+ return false;
+ }
+
+ *waiting = has_waiting;
+ return true;
+}
+
static void virtio_net_device_realize(DeviceState *dev, Error **errp)
{
VirtIODevice *vdev = VIRTIO_DEVICE(dev);
@@ -4001,6 +4129,12 @@ static void virtio_net_device_realize(DeviceState *dev, Error **errp)
n->nic->ncs[i].do_not_pad = true;
}
+ if (!virtio_net_check_peers_wait_incoming(n, &n->peers_wait_incoming,
+ errp)) {
+ virtio_cleanup(vdev);
+ return;
+ }
+
peer_test_vnet_hdr(n);
if (peer_has_vnet_hdr(n)) {
n->host_hdr_len = sizeof(struct virtio_net_hdr);
@@ -4310,6 +4444,7 @@ static const Property virtio_net_properties[] = {
host_features_ex,
VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO_CSUM,
true),
+ DEFINE_PROP_BOOL("local-migration", VirtIONet, local_migration, true),
};
static void virtio_net_class_init(ObjectClass *klass, const void *data)
diff --git a/include/hw/virtio/virtio-net.h b/include/hw/virtio/virtio-net.h
index 371e3764282..0c14e314409 100644
--- a/include/hw/virtio/virtio-net.h
+++ b/include/hw/virtio/virtio-net.h
@@ -230,6 +230,8 @@ struct VirtIONet {
struct EBPFRSSContext ebpf_rss;
uint32_t nr_ebpf_rss_fds;
char **ebpf_rss_fds;
+ bool peers_wait_incoming;
+ bool local_migration;
};
size_t virtio_net_handle_ctrl_iov(VirtIODevice *vdev,
diff --git a/include/net/net.h b/include/net/net.h
index aa34043b1ac..d4cf399d4a8 100644
--- a/include/net/net.h
+++ b/include/net/net.h
@@ -82,6 +82,7 @@ typedef void (SocketReadStateFinalize)(SocketReadState *rs);
typedef void (NetAnnounce)(NetClientState *);
typedef bool (SetSteeringEBPF)(NetClientState *, int);
typedef bool (NetCheckPeerType)(NetClientState *, ObjectClass *, Error **);
+typedef bool (IsWaitIncoming)(NetClientState *);
typedef struct vhost_net *(GetVHostNet)(NetClientState *nc);
typedef struct NetClientInfo {
@@ -110,6 +111,7 @@ typedef struct NetClientInfo {
NetAnnounce *announce;
SetSteeringEBPF *set_steering_ebpf;
NetCheckPeerType *check_peer_type;
+ IsWaitIncoming *is_wait_incoming;
GetVHostNet *get_vhost_net;
const VMStateDescription *backend_vmsd;
} NetClientInfo;
--
2.52.0
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [PATCH v16 6/8] net/tap: support local migration with virtio-net
2026-05-22 12:05 [PATCH v16 0/8] virtio-net: live-TAP local migration Vladimir Sementsov-Ogievskiy
` (4 preceding siblings ...)
2026-05-22 12:05 ` [PATCH v16 5/8] virtio-net: support local migration of backend Vladimir Sementsov-Ogievskiy
@ 2026-05-22 12:05 ` Vladimir Sementsov-Ogievskiy
2026-05-22 12:05 ` [PATCH v16 7/8] tests/functional: add skipWithoutSudo() decorator Vladimir Sementsov-Ogievskiy
` (2 subsequent siblings)
8 siblings, 0 replies; 25+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2026-05-22 12:05 UTC (permalink / raw)
To: jasowang, mst
Cc: armbru, peterx, farosas, raphael.s.norwitz, bchaney, vsementsov,
qemu-devel, berrange, pbonzini, yc-core, Eric Blake
Support transferring of TAP state (including open fd) through
migration stream as part of viritio-net "local-migration".
Add new option, incoming-fds, which should be set to true to
trigger new logic.
For new option require explicitly unset script and downscript,
to keep possibility of implementing support for them in future.
Note disabling read polling on source stop for TAP migration:
otherwise, source process may steal packages from TAP fd even
after source vm STOP.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Acked-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Ben Chaney <bchaney@akamai.com>
---
net/tap.c | 147 +++++++++++++++++++++++++++++++++++++++++++++++---
qapi/net.json | 10 +++-
2 files changed, 150 insertions(+), 7 deletions(-)
diff --git a/net/tap.c b/net/tap.c
index 9d6213fc3e5..9b1d4613a0b 100644
--- a/net/tap.c
+++ b/net/tap.c
@@ -36,6 +36,7 @@
#include "net/net.h"
#include "clients.h"
#include "monitor/monitor.h"
+#include "system/runstate.h"
#include "system/system.h"
#include "qapi/error.h"
#include "qemu/cutils.h"
@@ -86,6 +87,9 @@ typedef struct TAPState {
VHostNetState *vhost_net;
unsigned host_vnet_hdr_len;
Notifier exit;
+
+ bool read_poll_detached;
+ VMChangeStateEntry *vmstate;
} TAPState;
static void launch_script(const char *setup_script, const char *ifname,
@@ -94,19 +98,25 @@ static void launch_script(const char *setup_script, const char *ifname,
static void tap_send(void *opaque);
static void tap_writable(void *opaque);
+static bool tap_is_explicit_no_script(const char *script_arg)
+{
+ return script_arg &&
+ (script_arg[0] == '\0' || strcmp(script_arg, "no") == 0);
+}
+
static char *tap_parse_script(const char *script_arg, const char *default_path)
{
g_autofree char *res = g_strdup(script_arg);
- if (!res) {
- res = get_relocated_path(default_path);
+ if (tap_is_explicit_no_script(script_arg)) {
+ return NULL;
}
- if (res[0] == '\0' || strcmp(res, "no") == 0) {
- return NULL;
+ if (!script_arg) {
+ return get_relocated_path(default_path);
}
- return g_steal_pointer(&res);
+ return g_strdup(script_arg);
}
static void tap_update_fd_handler(TAPState *s)
@@ -123,6 +133,23 @@ static void tap_read_poll(TAPState *s, bool enable)
tap_update_fd_handler(s);
}
+static void tap_vm_state_change(void *opaque, bool running, RunState state)
+{
+ TAPState *s = opaque;
+
+ if (running) {
+ if (s->read_poll_detached) {
+ tap_read_poll(s, true);
+ s->read_poll_detached = false;
+ }
+ } else if (state == RUN_STATE_FINISH_MIGRATE) {
+ if (s->read_poll) {
+ s->read_poll_detached = true;
+ tap_read_poll(s, false);
+ }
+ }
+}
+
static void tap_write_poll(TAPState *s, bool enable)
{
s->write_poll = enable;
@@ -353,6 +380,11 @@ static void tap_cleanup(NetClientState *nc)
s->exit.notify = NULL;
}
+ if (s->vmstate) {
+ qemu_del_vm_change_state_handler(s->vmstate);
+ s->vmstate = NULL;
+ }
+
tap_read_poll(s, false);
tap_write_poll(s, false);
close(s->fd);
@@ -393,6 +425,65 @@ static VHostNetState *tap_get_vhost_net(NetClientState *nc)
return s->vhost_net;
}
+static bool tap_is_wait_incoming(NetClientState *nc)
+{
+ TAPState *s = DO_UPCAST(TAPState, nc, nc);
+ assert(nc->info->type == NET_CLIENT_DRIVER_TAP);
+ return s->fd == -1;
+}
+
+static int tap_pre_load(void *opaque)
+{
+ TAPState *s = opaque;
+
+ if (s->fd != -1) {
+ error_report(
+ "TAP is already initialized and cannot receive incoming fd");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static bool tap_setup_vhost(TAPState *s, Error **errp);
+
+static int tap_post_load(void *opaque, int version_id)
+{
+ TAPState *s = opaque;
+ Error *local_err = NULL;
+
+ tap_read_poll(s, true);
+
+ if (s->fd < 0) {
+ return -1;
+ }
+
+ if (!tap_setup_vhost(s, &local_err)) {
+ error_prepend(&local_err,
+ "Failed to setup vhost during TAP post-load: ");
+ error_report_err(local_err);
+ return -1;
+ }
+
+ return 0;
+}
+
+static const VMStateDescription vmstate_tap = {
+ .name = "net-tap",
+ .pre_load = tap_pre_load,
+ .post_load = tap_post_load,
+ .fields = (const VMStateField[]) {
+ VMSTATE_FD(fd, TAPState),
+ VMSTATE_BOOL(using_vnet_hdr, TAPState),
+ VMSTATE_BOOL(has_ufo, TAPState),
+ VMSTATE_BOOL(has_uso, TAPState),
+ VMSTATE_BOOL(has_tunnel, TAPState),
+ VMSTATE_BOOL(enabled, TAPState),
+ VMSTATE_UINT32(host_vnet_hdr_len, TAPState),
+ VMSTATE_END_OF_LIST()
+ }
+};
+
/* fd support */
static NetClientInfo net_tap_info = {
@@ -412,7 +503,9 @@ static NetClientInfo net_tap_info = {
.set_vnet_le = tap_set_vnet_le,
.set_vnet_be = tap_set_vnet_be,
.set_steering_ebpf = tap_set_steering_ebpf,
+ .is_wait_incoming = tap_is_wait_incoming,
.get_vhost_net = tap_get_vhost_net,
+ .backend_vmsd = &vmstate_tap,
};
static TAPState *net_tap_fd_init(NetClientState *peer,
@@ -748,6 +841,9 @@ static bool net_init_tap_one(const NetdevTapOptions *tap, NetClientState *peer,
int sndbuf =
(tap->has_sndbuf && tap->sndbuf) ? MIN(tap->sndbuf, INT_MAX) : INT_MAX;
+ s->read_poll_detached = false;
+ s->vmstate = qemu_add_vm_change_state_handler(tap_vm_state_change, s);
+
if (!tap_set_sndbuf(fd, sndbuf, sndbuf_required ? errp : NULL) &&
sndbuf_required) {
goto failed;
@@ -779,6 +875,8 @@ static bool net_init_tap_one(const NetdevTapOptions *tap, NetClientState *peer,
return true;
failed:
+ qemu_del_vm_change_state_handler(s->vmstate);
+ s->vmstate = NULL;
qemu_del_net_client(&s->nc);
return false;
}
@@ -910,6 +1008,26 @@ int net_init_tap(const Netdev *netdev, const char *name,
return -1;
}
+ if (tap->incoming_fds &&
+ (tap->fd || tap->fds || tap->helper || tap->br || tap->ifname ||
+ tap->has_sndbuf || tap->has_vnet_hdr)) {
+ error_setg(errp, "incoming-fds is incompatible with "
+ "fd=, fds=, helper=, br=, ifname=, sndbuf= and vnet_hdr=");
+ return -1;
+ }
+
+ if (tap->incoming_fds &&
+ !(tap_is_explicit_no_script(tap->script) &&
+ tap_is_explicit_no_script(tap->downscript))) {
+ /*
+ * script="" and downscript="" are silently supported to be consistent
+ * with cases without incoming_fds, but do not care to put this into
+ * error message.
+ */
+ error_setg(errp, "incoming-fds requires script=no and downscript=no");
+ return -1;
+ }
+
queues = tap_parse_fds_and_queues(tap, &fds, errp);
if (queues < 0) {
return -1;
@@ -928,7 +1046,24 @@ int net_init_tap(const Netdev *netdev, const char *name,
goto fail;
}
- if (fds) {
+ if (tap->incoming_fds) {
+ for (i = 0; i < queues; i++) {
+ NetClientState *nc;
+ TAPState *s;
+
+ nc = qemu_new_net_client(&net_tap_info, peer, "tap", name);
+ qemu_set_info_str(nc, "incoming");
+
+ s = DO_UPCAST(TAPState, nc, nc);
+ s->fd = -1;
+ if (vhost_fds) {
+ s->vhostfd = vhost_fds[i];
+ s->vhost_busyloop_timeout = tap->has_poll_us ? tap->poll_us : 0;
+ } else {
+ s->vhostfd = -1;
+ }
+ }
+ } else if (fds) {
for (i = 0; i < queues; i++) {
if (i == 0) {
vnet_hdr = tap_probe_vnet_hdr(fds[i], errp);
diff --git a/qapi/net.json b/qapi/net.json
index 1a6382825c5..82ddbb51cd7 100644
--- a/qapi/net.json
+++ b/qapi/net.json
@@ -425,6 +425,13 @@
# @poll-us: maximum number of microseconds that could be spent on busy
# polling for tap (since 2.7)
#
+# @incoming-fds: do not open or create any TAP devices. Prepare for
+# getting TAP file descriptors from incoming migration stream.
+# The option is incompatible with any of @fd, @fds, @helper, @br,
+# @ifname, @sndbuf and @vnet_hdr options, and requires @script and
+# @downscript be explicitly set to nothing (empty string or "no")
+# (Since 11.1)
+#
# Since: 1.2
##
{ 'struct': 'NetdevTapOptions',
@@ -443,7 +450,8 @@
'*vhostfds': 'str',
'*vhostforce': 'bool',
'*queues': 'uint32',
- '*poll-us': 'uint32'} }
+ '*poll-us': 'uint32',
+ '*incoming-fds': 'bool' } }
##
# @NetdevSocketOptions:
--
2.52.0
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [PATCH v16 7/8] tests/functional: add skipWithoutSudo() decorator
2026-05-22 12:05 [PATCH v16 0/8] virtio-net: live-TAP local migration Vladimir Sementsov-Ogievskiy
` (5 preceding siblings ...)
2026-05-22 12:05 ` [PATCH v16 6/8] net/tap: support local migration with virtio-net Vladimir Sementsov-Ogievskiy
@ 2026-05-22 12:05 ` Vladimir Sementsov-Ogievskiy
2026-05-22 12:05 ` [PATCH v16 8/8] tests/functional: add test_tap_migration Vladimir Sementsov-Ogievskiy
2026-05-27 10:46 ` [PATCH v16 0/8] virtio-net: live-TAP local migration Mark Cave-Ayland
8 siblings, 0 replies; 25+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2026-05-22 12:05 UTC (permalink / raw)
To: jasowang, mst
Cc: armbru, peterx, farosas, raphael.s.norwitz, bchaney, vsementsov,
qemu-devel, berrange, pbonzini, yc-core, Thomas Huth, Lei Yang,
Maksim Davydov, Thomas Huth, Philippe Mathieu-Daudé
To be used in the next commit: that would be a test for TAP
networking, and it will need to setup TAP device.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Tested-by: Lei Yang <leiyang@redhat.com>
Reviewed-by: Maksim Davydov <davydov-max@yandex-team.ru>
Reviewed-by: Ben Chaney <bchaney@akamai.com>
---
tests/functional/qemu_test/decorators.py | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/tests/functional/qemu_test/decorators.py b/tests/functional/qemu_test/decorators.py
index fcf236ecfdf..aa135acc785 100644
--- a/tests/functional/qemu_test/decorators.py
+++ b/tests/functional/qemu_test/decorators.py
@@ -6,6 +6,7 @@
import os
import platform
import resource
+import subprocess
from unittest import skipIf, skipUnless
from .cmd import which
@@ -177,3 +178,18 @@ def skipLockedMemoryTest(locked_memory):
ulimit_memory == resource.RLIM_INFINITY or ulimit_memory >= locked_memory * 1024,
f'Test required {locked_memory} kB of available locked memory',
)
+
+'''
+Decorator to skip execution of a test if passwordless
+sudo command is not available.
+'''
+def skipWithoutSudo():
+ proc = subprocess.run(["sudo", "-n", "/bin/true"],
+ stdin=subprocess.PIPE,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT,
+ universal_newlines=True,
+ check=False)
+
+ return skipUnless(proc.returncode == 0,
+ f'requires password-less sudo access: {proc.stdout}')
--
2.52.0
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [PATCH v16 8/8] tests/functional: add test_tap_migration
2026-05-22 12:05 [PATCH v16 0/8] virtio-net: live-TAP local migration Vladimir Sementsov-Ogievskiy
` (6 preceding siblings ...)
2026-05-22 12:05 ` [PATCH v16 7/8] tests/functional: add skipWithoutSudo() decorator Vladimir Sementsov-Ogievskiy
@ 2026-05-22 12:05 ` Vladimir Sementsov-Ogievskiy
2026-05-27 10:46 ` [PATCH v16 0/8] virtio-net: live-TAP local migration Mark Cave-Ayland
8 siblings, 0 replies; 25+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2026-05-22 12:05 UTC (permalink / raw)
To: jasowang, mst
Cc: armbru, peterx, farosas, raphael.s.norwitz, bchaney, vsementsov,
qemu-devel, berrange, pbonzini, yc-core, Zhao Liu
Add test for a new local-migration migration of virtio-net/tap, with fd
passing through UNIX socket.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Reviewed-by: Ben Chaney <bchaney@akamai.com>
---
tests/functional/x86_64/meson.build | 1 +
tests/functional/x86_64/test_tap_migration.py | 458 ++++++++++++++++++
2 files changed, 459 insertions(+)
create mode 100755 tests/functional/x86_64/test_tap_migration.py
diff --git a/tests/functional/x86_64/meson.build b/tests/functional/x86_64/meson.build
index 1ed10ad6c29..c4bb14f034b 100644
--- a/tests/functional/x86_64/meson.build
+++ b/tests/functional/x86_64/meson.build
@@ -45,4 +45,5 @@ tests_x86_64_system_thorough = [
'virtio_balloon',
'virtio_gpu',
'rebuild_vmfd',
+ 'tap_migration',
]
diff --git a/tests/functional/x86_64/test_tap_migration.py b/tests/functional/x86_64/test_tap_migration.py
new file mode 100755
index 00000000000..7708facc153
--- /dev/null
+++ b/tests/functional/x86_64/test_tap_migration.py
@@ -0,0 +1,458 @@
+#!/usr/bin/env python3
+#
+# Functional test that tests TAP local migration
+# with fd passing
+#
+# Copyright (c) Yandex Technologies LLC, 2026
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+
+import os
+import time
+import subprocess
+from subprocess import run
+import signal
+import ctypes
+import ctypes.util
+import unittest
+from contextlib import contextmanager, ExitStack
+from typing import Tuple
+
+from qemu_test import (
+ LinuxKernelTest,
+ Asset,
+ exec_command_and_wait_for_pattern,
+)
+from qemu_test.decorators import skipWithoutSudo
+
+
+GUEST_IP = "192.168.100.2"
+GUEST_IP_MASK = f"{GUEST_IP}/24"
+GUEST_MAC = "d6:0d:75:f8:0f:b7"
+HOST_IP = "192.168.100.1"
+HOST_IP_MASK = f"{HOST_IP}/24"
+TAP_ID = "tap0"
+TAP_ID2 = "tap1"
+TAP_MAC = "e6:1d:44:b5:03:5d"
+NETNS = f"qemu_test_ns_{os.getpid()}"
+
+
+def ip(args, check=True) -> None:
+ """Run ip command with sudo"""
+ run(["sudo", "ip"] + args, check=check)
+
+
+@contextmanager
+def switch_netns(netns_name):
+ libc = ctypes.CDLL(ctypes.util.find_library("c"))
+ netns_path = f"/var/run/netns/{netns_name}"
+
+ def switch_to_fd(fd, check: bool = False):
+ """Switch to netns by file descriptor"""
+ SYS_setns = 308
+ CLONE_NEWNET = 0x40000000
+ ret = libc.syscall(SYS_setns, fd, CLONE_NEWNET)
+ if check and ret != 0:
+ raise RuntimeError("syscall SETNS failed")
+
+ with ExitStack() as stack:
+ original_netns_fd = os.open("/proc/self/ns/net", os.O_RDONLY)
+ stack.callback(os.close, original_netns_fd)
+
+ ip(["netns", "add", netns_name])
+ stack.callback(ip, ["netns", "del", netns_name], check=False)
+
+ new_netns_fd = os.open(netns_path, os.O_RDONLY)
+ stack.callback(os.close, new_netns_fd)
+
+ switch_to_fd(new_netns_fd)
+ stack.callback(switch_to_fd, original_netns_fd, check=False)
+
+ yield
+
+
+def del_tap(tap_name: str = TAP_ID) -> None:
+ ip(["tuntap", "del", tap_name, "mode", "tap", "multi_queue"], check=False)
+
+
+def init_tap(tap_name: str = TAP_ID, with_ip: bool = True) -> None:
+ ip(["tuntap", "add", "dev", tap_name, "mode", "tap", "multi_queue"])
+ if with_ip:
+ ip(["link", "set", "dev", tap_name, "address", TAP_MAC])
+ ip(["addr", "add", HOST_IP_MASK, "dev", tap_name])
+ ip(["link", "set", tap_name, "up"])
+
+
+def switch_network_to_tap2() -> None:
+ ip(["link", "set", TAP_ID2, "down"])
+ ip(["link", "set", TAP_ID, "down"])
+ ip(["addr", "delete", HOST_IP_MASK, "dev", TAP_ID])
+ ip(["link", "set", "dev", TAP_ID2, "address", TAP_MAC])
+ ip(["addr", "add", HOST_IP_MASK, "dev", TAP_ID2])
+ ip(["link", "set", TAP_ID2, "up"])
+
+
+def parse_ping_line(line: str) -> float:
+ # suspect lines like
+ # [1748524876.590509] 64 bytes from 94.245.155.3 \
+ # (94.245.155.3): icmp_seq=1 ttl=250 time=101 ms
+ spl = line.split()
+ return float(spl[0][1:-1])
+
+
+def parse_ping_output(out) -> Tuple[bool, float, float]:
+ lines = [x for x in out.split("\n") if x.startswith("[")]
+
+ try:
+ first_no_ans = next(
+ (ind for ind in range(len(lines)) if lines[ind][20:26] == "no ans")
+ )
+ except StopIteration:
+ return False, parse_ping_line(lines[0]), parse_ping_line(lines[-1])
+
+ last_no_ans = next(
+ ind
+ for ind in range(len(lines) - 1, -1, -1)
+ if lines[ind][20:26] == "no ans"
+ )
+
+ return (
+ True,
+ parse_ping_line(lines[first_no_ans]),
+ parse_ping_line(lines[last_no_ans]),
+ )
+
+
+def wait_migration_finish(source_vm, target_vm):
+ migr_events = (
+ ("MIGRATION", {"data": {"status": "completed"}}),
+ ("MIGRATION", {"data": {"status": "failed"}}),
+ )
+
+ source_e = source_vm.events_wait(migr_events)["data"]
+ target_e = target_vm.events_wait(migr_events)["data"]
+
+ source_s = source_vm.cmd("query-status")["status"]
+ target_s = target_vm.cmd("query-status")["status"]
+
+ assert (
+ source_e["status"] == "completed"
+ and target_e["status"] == "completed"
+ and source_s == "postmigrate"
+ and target_s == "paused"
+ ), f"""Migration failed:
+ SRC status: {source_s}
+ SRC event: {source_e}
+ TGT status: {target_s}
+ TGT event:{target_e}"""
+
+
+@skipWithoutSudo()
+class TAPFdMigration(LinuxKernelTest):
+
+ ASSET_KERNEL = Asset(
+ (
+ "https://archives.fedoraproject.org/pub/archive/fedora/linux/releases"
+ "/31/Server/x86_64/os/images/pxeboot/vmlinuz"
+ ),
+ "d4738d03dbbe083ca610d0821d0a8f1488bebbdccef54ce33e3adb35fda00129",
+ )
+
+ ASSET_INITRD = Asset(
+ (
+ "https://archives.fedoraproject.org/pub/archive/fedora/linux/releases"
+ "/31/Server/x86_64/os/images/pxeboot/initrd.img"
+ ),
+ "277cd6c7adf77c7e63d73bbb2cded8ef9e2d3a2f100000e92ff1f8396513cd8b",
+ )
+
+ ASSET_ALPINE_ISO = Asset(
+ (
+ "https://dl-cdn.alpinelinux.org/"
+ "alpine/v3.22/releases/x86_64/alpine-standard-3.22.1-x86_64.iso"
+ ),
+ "96d1b44ea1b8a5a884f193526d92edb4676054e9fa903ad2f016441a0fe13089",
+ )
+
+ @classmethod
+ def setUpClass(cls):
+ super().setUpClass()
+
+ try:
+ cls.netns_context = switch_netns(NETNS)
+ cls.netns_context.__enter__()
+ except (OSError, subprocess.CalledProcessError) as e:
+ raise unittest.SkipTest(f"can't switch network namespace: {e}")
+
+ @classmethod
+ def tearDownClass(cls):
+ if hasattr(cls, "netns_context"):
+ cls.netns_context.__exit__(None, None, None)
+ super().tearDownClass()
+
+ def setUp(self):
+ super().setUp()
+
+ init_tap()
+
+ self.outer_ping_proc = None
+ self.shm_path = None
+
+ def tearDown(self):
+ try:
+ del_tap(TAP_ID)
+ del_tap(TAP_ID2)
+
+ if self.outer_ping_proc:
+ self.stop_outer_ping()
+
+ if self.shm_path:
+ os.unlink(self.shm_path)
+ finally:
+ super().tearDown()
+
+ def start_outer_ping(self) -> None:
+ assert self.outer_ping_proc is None
+ self.outer_ping_log = self.scratch_file("ping.log")
+ with open(self.outer_ping_log, "w") as f:
+ self.outer_ping_proc = subprocess.Popen(
+ ["ping", "-i", "0", "-O", "-D", GUEST_IP],
+ text=True,
+ stdout=f,
+ )
+
+ def stop_outer_ping(self) -> str:
+ assert self.outer_ping_proc
+ self.outer_ping_proc.send_signal(signal.SIGINT)
+
+ self.outer_ping_proc.communicate(timeout=5)
+ self.outer_ping_proc = None
+
+ with open(self.outer_ping_log) as f:
+ return f.read()
+
+ def stop_ping_and_check(self, stop_time, resume_time):
+ ping_res = self.stop_outer_ping()
+
+ discon, a, b = parse_ping_output(ping_res)
+
+ if not discon:
+ text = (
+ f"STOP: {stop_time}, RESUME: {resume_time}," f"PING: {a} - {b}"
+ )
+ if a > stop_time or b < resume_time:
+ self.fail(f"PING failed: {text}")
+ self.log.info(f"PING: no packets lost: {text}")
+ return
+
+ text = (
+ f"STOP: {stop_time}, RESUME: {resume_time},"
+ f"PING: disconnect: {a} - {b}"
+ )
+ self.log.info(text)
+ eps = 0.05
+ if a < stop_time - eps or b > resume_time + eps:
+ self.fail(text)
+
+ def one_ping_from_guest(self, vm) -> None:
+ exec_command_and_wait_for_pattern(
+ self,
+ f"ping -c 1 -W 1 {HOST_IP}",
+ "1 packets transmitted, 1 packets received",
+ "1 packets transmitted, 0 packets received",
+ vm=vm,
+ )
+ self.wait_for_console_pattern("# ", vm=vm)
+
+ def one_ping_from_host(self) -> None:
+ run(
+ ["ping", "-c", "1", "-W", "1", GUEST_IP],
+ stdout=subprocess.DEVNULL,
+ check=True,
+ )
+
+ def setup_shared_memory(self):
+ self.shm_path = f"/dev/shm/qemu_test_{os.getpid()}"
+
+ try:
+ with open(self.shm_path, "wb") as f:
+ f.write(b"\0" * (1024 * 1024 * 1024)) # 1GB
+ except Exception as e:
+ self.fail(f"Failed to create shared memory file: {e}")
+
+ def prepare_and_launch_vm(
+ self, shm_path, vhost, incoming=False, vm=None, local=True
+ ):
+ if not vm:
+ vm = self.vm
+
+ vm.set_console()
+ vm.add_args("-accel", "kvm")
+ vm.add_args("-device", "pcie-pci-bridge,id=pci.1,bus=pcie.0")
+ vm.add_args("-m", "1G")
+
+ vm.add_args(
+ "-object",
+ f"memory-backend-file,id=ram0,size=1G,mem-path={shm_path},share=on",
+ )
+ vm.add_args("-machine", "memory-backend=ram0")
+
+ vm.add_args(
+ "-drive",
+ f"file={self.ASSET_ALPINE_ISO.fetch()},media=cdrom,format=raw",
+ )
+
+ vm.add_args("-S")
+
+ if incoming:
+ vm.add_args("-incoming", "defer")
+
+ vm_s = "target" if incoming else "source"
+ self.log.info(f"Launching {vm_s} VM")
+ vm.launch()
+
+ if not local:
+ tap_name = TAP_ID2 if incoming else TAP_ID
+ else:
+ tap_name = TAP_ID
+
+ self.add_virtio_net(vm, vhost, tap_name, local, incoming)
+
+ self.set_migration_capabilities(vm, local)
+
+ def add_virtio_net(
+ self, vm, vhost: bool, tap_name: str, local: bool, incoming: bool
+ ):
+ incoming_fds = local and incoming
+ netdev_params = {
+ "id": "netdev.1",
+ "vhost": vhost,
+ "type": "tap",
+ "queues": 4,
+ "incoming-fds": incoming_fds,
+ "script": "no",
+ "downscript": "no",
+ }
+
+ if not incoming_fds:
+ netdev_params["vnet_hdr"] = True
+ netdev_params["ifname"] = tap_name
+
+ vm.cmd("netdev_add", netdev_params)
+
+ vm.cmd(
+ "device_add",
+ driver="virtio-net-pci",
+ romfile="",
+ id="vnet.1",
+ netdev="netdev.1",
+ mq=True,
+ vectors=18,
+ bus="pci.1",
+ mac=GUEST_MAC,
+ disable_legacy="off",
+ local_migration=local,
+ )
+
+ def set_migration_capabilities(self, vm, local=True):
+ vm.cmd(
+ "migrate-set-capabilities",
+ {
+ "capabilities": [
+ {"capability": "events", "state": True},
+ {"capability": "x-ignore-shared", "state": True},
+ ]
+ },
+ )
+ vm.cmd("migrate-set-parameters", {"local": local})
+
+ def setup_guest_network(self) -> None:
+ exec_command_and_wait_for_pattern(self, "ip addr", "# ")
+ exec_command_and_wait_for_pattern(
+ self,
+ f"ip addr add {GUEST_IP_MASK} dev eth0 && "
+ "ip link set eth0 up && echo OK",
+ "OK",
+ )
+ self.wait_for_console_pattern("# ")
+
+ def do_test_tap_fd_migration(self, vhost, local=True):
+ self.require_accelerator("kvm")
+ self.set_machine("q35")
+
+ socket_dir = self.socket_dir()
+ migration_socket = os.path.join(socket_dir.name, "migration.sock")
+
+ self.setup_shared_memory()
+
+ # Setup second TAP if needed
+ if not local:
+ del_tap(TAP_ID2)
+ init_tap(TAP_ID2, with_ip=False)
+
+ self.prepare_and_launch_vm(self.shm_path, vhost, local=local)
+ self.vm.cmd("cont")
+ self.wait_for_console_pattern("login:")
+ exec_command_and_wait_for_pattern(self, "root", "# ")
+
+ self.setup_guest_network()
+
+ self.one_ping_from_guest(self.vm)
+ self.one_ping_from_host()
+ self.start_outer_ping()
+
+ # Get some successful pings before migration
+ time.sleep(0.5)
+
+ target_vm = self.get_vm(name="target")
+ self.prepare_and_launch_vm(
+ self.shm_path,
+ vhost,
+ incoming=True,
+ vm=target_vm,
+ local=local,
+ )
+
+ target_vm.cmd("migrate-incoming", {"uri": f"unix:{migration_socket}"})
+
+ self.log.info("Starting migration")
+ freeze_start = time.time()
+ self.vm.cmd("migrate", {"uri": f"unix:{migration_socket}"})
+
+ self.log.info("Waiting for migration completion")
+ wait_migration_finish(self.vm, target_vm)
+
+ # Switch network to tap1 if not using local-migration
+ if not local:
+ switch_network_to_tap2()
+
+ target_vm.cmd("cont")
+ freeze_end = time.time()
+
+ self.vm.shutdown()
+
+ self.log.info("Verifying PING on target VM after migration")
+ self.one_ping_from_guest(target_vm)
+ self.one_ping_from_host()
+
+ # And a bit more pings after source shutdown
+ time.sleep(0.3)
+ self.stop_ping_and_check(freeze_start, freeze_end)
+
+ target_vm.shutdown()
+
+ def test_tap_fd_migration(self):
+ self.do_test_tap_fd_migration(False)
+
+ def test_tap_fd_migration_vhost(self):
+ self.do_test_tap_fd_migration(True)
+
+ def test_tap_new_tap_migration(self):
+ self.do_test_tap_fd_migration(False, local=False)
+
+ def test_tap_new_tap_migration_vhost(self):
+ self.do_test_tap_fd_migration(True, local=False)
+
+
+if __name__ == "__main__":
+ LinuxKernelTest.main()
--
2.52.0
^ permalink raw reply related [flat|nested] 25+ messages in thread
* Re: [PATCH v16 5/8] virtio-net: support local migration of backend
2026-05-22 12:05 ` [PATCH v16 5/8] virtio-net: support local migration of backend Vladimir Sementsov-Ogievskiy
@ 2026-05-24 9:09 ` Michael S. Tsirkin
2026-05-25 13:51 ` Vladimir Sementsov-Ogievskiy
0 siblings, 1 reply; 25+ messages in thread
From: Michael S. Tsirkin @ 2026-05-24 9:09 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy
Cc: jasowang, armbru, peterx, farosas, raphael.s.norwitz, bchaney,
qemu-devel, berrange, pbonzini, yc-core,
Philippe Mathieu-Daudé, Zhao Liu, Richard Henderson
On Fri, May 22, 2026 at 03:05:30PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> Add virtio-net option local-migration, which is true by default,
> but false for older machine types, which doesn't support the feature.
>
> When both global migration parameter "local" and new virtio-net
> parameter "local-migration" are true, virtio-net transfer the whole
> net backend to the destination, including open file descriptors.
> Of-course, its only for local migration and the channel must be
> UNIX domain socket.
>
> This way management tool should not care about creating new TAP, and
> should not handle switching to it. Migration downtime become shorter.
>
> Support for TAP will come in the next commit.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
> Reviewed-by: Ben Chaney <bchaney@akamai.com>
I don't get why is this a device property?
It's clearly a backend thing?
> ---
> hw/core/machine.c | 1 +
> hw/i386/pc_q35.c | 1 +
> hw/net/virtio-net.c | 137 ++++++++++++++++++++++++++++++++-
> include/hw/virtio/virtio-net.h | 2 +
> include/net/net.h | 2 +
> 5 files changed, 142 insertions(+), 1 deletion(-)
>
> diff --git a/hw/core/machine.c b/hw/core/machine.c
> index 63baff859f3..619e80c1cb3 100644
> --- a/hw/core/machine.c
> +++ b/hw/core/machine.c
> @@ -41,6 +41,7 @@
>
> GlobalProperty hw_compat_11_0[] = {
> { "chardev-vc", "encoding", "cp437" },
> + { TYPE_VIRTIO_NET, "local-migration", "false" },
> };
> const size_t hw_compat_11_0_len = G_N_ELEMENTS(hw_compat_11_0);
>
> diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
> index d8fed698c72..b5c0e302d59 100644
> --- a/hw/i386/pc_q35.c
> +++ b/hw/i386/pc_q35.c
> @@ -368,6 +368,7 @@ static void pc_q35_machine_options(MachineClass *m)
> static void pc_q35_machine_11_1_options(MachineClass *m)
> {
> pc_q35_machine_options(m);
> + compat_props_add(m->compat_props, hw_compat_11_0, hw_compat_11_0_len);
> }
>
> DEFINE_Q35_MACHINE_AS_LATEST(11, 1);
> diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
> index 2a5d642a647..158b9247a58 100644
> --- a/hw/net/virtio-net.c
> +++ b/hw/net/virtio-net.c
> @@ -38,8 +38,10 @@
> #include "qapi/qapi-events-migration.h"
> #include "hw/virtio/virtio-access.h"
> #include "migration/misc.h"
> +#include "migration/options.h"
> #include "standard-headers/linux/ethtool.h"
> #include "system/system.h"
> +#include "system/runstate.h"
> #include "system/replay.h"
> #include "trace.h"
> #include "monitor/qdev.h"
> @@ -3060,7 +3062,17 @@ static void virtio_net_set_multiqueue(VirtIONet *n, int multiqueue)
> n->multiqueue = multiqueue;
> virtio_net_change_num_queues(n, max * 2 + 1);
>
> - virtio_net_set_queue_pairs(n);
> + /*
> + * virtio_net_set_multiqueue() called from set_features(0) on early
> + * reset, when peer may wait for incoming (and is not initialized
> + * yet).
> + * Don't worry about it: virtio_net_set_queue_pairs() will be called
> + * later form virtio_net_post_load_device(), and anyway will be
> + * noop for local incoming migration with live backend passing.
> + */
> + if (!n->peers_wait_incoming) {
> + virtio_net_set_queue_pairs(n);
> + }
> }
>
> static int virtio_net_pre_load_queues(VirtIODevice *vdev, uint32_t n)
> @@ -3089,6 +3101,17 @@ static void virtio_net_get_features(VirtIODevice *vdev, uint64_t *features,
>
> virtio_add_feature_ex(features, VIRTIO_NET_F_MAC);
>
> + if (n->peers_wait_incoming) {
> + /*
> + * Excessive feature set is OK for early initialization when
> + * we wait for local incoming migration: actual guest-negotiated
> + * features will come with migration stream anyway. And we are sure
> + * that we support same host-features as source, because the backend
> + * is the same (the same TAP device, for example).
> + */
> + return;
> + }
> +
> if (!peer_has_vnet_hdr(n)) {
> virtio_clear_feature_ex(features, VIRTIO_NET_F_CSUM);
> virtio_clear_feature_ex(features, VIRTIO_NET_F_HOST_TSO4);
> @@ -3179,6 +3202,18 @@ static void virtio_net_get_features(VirtIODevice *vdev, uint64_t *features,
> }
> }
>
> +static bool virtio_net_update_host_features(VirtIONet *n, Error **errp)
> +{
> + ERRP_GUARD();
> + VirtIODevice *vdev = VIRTIO_DEVICE(n);
> +
> + peer_test_vnet_hdr(n);
> +
> + virtio_net_get_features(vdev, &vdev->host_features, errp);
> +
> + return !*errp;
> +}
> +
> static int virtio_net_post_load_device(void *opaque, int version_id)
> {
> VirtIONet *n = opaque;
> @@ -3300,6 +3335,9 @@ struct VirtIONetMigTmp {
> uint16_t curr_queue_pairs_1;
> uint8_t has_ufo;
> uint32_t has_vnet_hdr;
> +
> + NetClientState *ncs;
> + uint32_t max_queue_pairs;
> };
>
> /* The 2nd and subsequent tx_waiting flags are loaded later than
> @@ -3569,6 +3607,57 @@ static const VMStateDescription vhost_user_net_backend_state = {
> }
> };
>
> +static bool virtio_net_migrate_local(void *opaque, int version_id)
> +{
> + VirtIONet *n = opaque;
> +
> + return migrate_local() && n->local_migration;
> +}
> +
> +static int virtio_net_nic_pre_save(void *opaque)
> +{
> + struct VirtIONetMigTmp *tmp = opaque;
> +
> + tmp->ncs = tmp->parent->nic->ncs;
> + tmp->max_queue_pairs = tmp->parent->max_queue_pairs;
> +
> + return 0;
> +}
> +
> +static int virtio_net_nic_pre_load(void *opaque)
> +{
> + /* Reuse the pointer setup from save */
> + virtio_net_nic_pre_save(opaque);
> +
> + return 0;
> +}
> +
> +static int virtio_net_nic_post_load(void *opaque, int version_id)
> +{
> + struct VirtIONetMigTmp *tmp = opaque;
> + Error *local_err = NULL;
> +
> + if (!virtio_net_update_host_features(tmp->parent, &local_err)) {
> + error_report_err(local_err);
> + return -EINVAL;
> + }
> +
> + return 0;
> +}
> +
> +static const VMStateDescription vmstate_virtio_net_nic = {
> + .name = "virtio-net-nic",
> + .pre_load = virtio_net_nic_pre_load,
> + .pre_save = virtio_net_nic_pre_save,
> + .post_load = virtio_net_nic_post_load,
> + .fields = (const VMStateField[]) {
> + VMSTATE_VARRAY_UINT32(ncs, struct VirtIONetMigTmp,
> + max_queue_pairs, 0, vmstate_net_peer_backend,
> + NetClientState),
> + VMSTATE_END_OF_LIST()
> + },
> +};
> +
> static const VMStateDescription vmstate_virtio_net_device = {
> .name = "virtio-net-device",
> .version_id = VIRTIO_NET_VM_VERSION,
> @@ -3600,6 +3689,9 @@ static const VMStateDescription vmstate_virtio_net_device = {
> * but based on the uint.
> */
> VMSTATE_BUFFER_POINTER_UNSAFE(vlans, VirtIONet, 0, MAX_VLAN >> 3),
> + VMSTATE_WITH_TMP_TEST(VirtIONet, virtio_net_migrate_local,
> + struct VirtIONetMigTmp,
> + vmstate_virtio_net_nic),
> VMSTATE_WITH_TMP(VirtIONet, struct VirtIONetMigTmp,
> vmstate_virtio_net_has_vnet),
> VMSTATE_UINT8(mac_table.multi_overflow, VirtIONet),
> @@ -3864,6 +3956,42 @@ static bool failover_hide_primary_device(DeviceListener *listener,
> return qatomic_read(&n->failover_primary_hidden);
> }
>
> +static bool virtio_net_check_peers_wait_incoming(VirtIONet *n, bool *waiting,
> + Error **errp)
> +{
> + bool has_waiting = false;
> + bool has_not_waiting = false;
> +
> + for (int i = 0; i < n->max_queue_pairs; i++) {
> + NetClientState *peer = n->nic->ncs[i].peer;
> + if (!peer) {
> + continue;
> + }
> +
> + if (peer->info->is_wait_incoming &&
> + peer->info->is_wait_incoming(peer)) {
> + has_waiting = true;
> + } else {
> + has_not_waiting = true;
> + }
> +
> + if (has_waiting && has_not_waiting) {
> + error_setg(errp, "Mixed peer states: some peers wait for incoming "
> + "migration while others don't");
> + return false;
> + }
> + }
> +
> + if (has_waiting && !runstate_check(RUN_STATE_INMIGRATE)) {
> + error_setg(errp, "Peers wait for incoming, but it's not an incoming "
> + "migration.");
> + return false;
> + }
> +
> + *waiting = has_waiting;
> + return true;
> +}
> +
> static void virtio_net_device_realize(DeviceState *dev, Error **errp)
> {
> VirtIODevice *vdev = VIRTIO_DEVICE(dev);
> @@ -4001,6 +4129,12 @@ static void virtio_net_device_realize(DeviceState *dev, Error **errp)
> n->nic->ncs[i].do_not_pad = true;
> }
>
> + if (!virtio_net_check_peers_wait_incoming(n, &n->peers_wait_incoming,
> + errp)) {
> + virtio_cleanup(vdev);
> + return;
> + }
> +
> peer_test_vnet_hdr(n);
> if (peer_has_vnet_hdr(n)) {
> n->host_hdr_len = sizeof(struct virtio_net_hdr);
> @@ -4310,6 +4444,7 @@ static const Property virtio_net_properties[] = {
> host_features_ex,
> VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO_CSUM,
> true),
> + DEFINE_PROP_BOOL("local-migration", VirtIONet, local_migration, true),
> };
>
> static void virtio_net_class_init(ObjectClass *klass, const void *data)
> diff --git a/include/hw/virtio/virtio-net.h b/include/hw/virtio/virtio-net.h
> index 371e3764282..0c14e314409 100644
> --- a/include/hw/virtio/virtio-net.h
> +++ b/include/hw/virtio/virtio-net.h
> @@ -230,6 +230,8 @@ struct VirtIONet {
> struct EBPFRSSContext ebpf_rss;
> uint32_t nr_ebpf_rss_fds;
> char **ebpf_rss_fds;
> + bool peers_wait_incoming;
> + bool local_migration;
> };
>
> size_t virtio_net_handle_ctrl_iov(VirtIODevice *vdev,
> diff --git a/include/net/net.h b/include/net/net.h
> index aa34043b1ac..d4cf399d4a8 100644
> --- a/include/net/net.h
> +++ b/include/net/net.h
> @@ -82,6 +82,7 @@ typedef void (SocketReadStateFinalize)(SocketReadState *rs);
> typedef void (NetAnnounce)(NetClientState *);
> typedef bool (SetSteeringEBPF)(NetClientState *, int);
> typedef bool (NetCheckPeerType)(NetClientState *, ObjectClass *, Error **);
> +typedef bool (IsWaitIncoming)(NetClientState *);
> typedef struct vhost_net *(GetVHostNet)(NetClientState *nc);
>
> typedef struct NetClientInfo {
> @@ -110,6 +111,7 @@ typedef struct NetClientInfo {
> NetAnnounce *announce;
> SetSteeringEBPF *set_steering_ebpf;
> NetCheckPeerType *check_peer_type;
> + IsWaitIncoming *is_wait_incoming;
> GetVHostNet *get_vhost_net;
> const VMStateDescription *backend_vmsd;
> } NetClientInfo;
> --
> 2.52.0
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v16 5/8] virtio-net: support local migration of backend
2026-05-24 9:09 ` Michael S. Tsirkin
@ 2026-05-25 13:51 ` Vladimir Sementsov-Ogievskiy
2026-05-25 14:45 ` Peter Xu
0 siblings, 1 reply; 25+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2026-05-25 13:51 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: jasowang, armbru, peterx, farosas, raphael.s.norwitz, bchaney,
qemu-devel, berrange, pbonzini, yc-core,
Philippe Mathieu-Daudé, Zhao Liu, Richard Henderson
On 24.05.26 12:09, Michael S. Tsirkin wrote:
> On Fri, May 22, 2026 at 03:05:30PM +0300, Vladimir Sementsov-Ogievskiy wrote:
>> Add virtio-net option local-migration, which is true by default,
>> but false for older machine types, which doesn't support the feature.
>>
>> When both global migration parameter "local" and new virtio-net
>> parameter "local-migration" are true, virtio-net transfer the whole
>> net backend to the destination, including open file descriptors.
>> Of-course, its only for local migration and the channel must be
>> UNIX domain socket.
>>
>> This way management tool should not care about creating new TAP, and
>> should not handle switching to it. Migration downtime become shorter.
>>
>> Support for TAP will come in the next commit.
>>
>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
>> Reviewed-by: Ben Chaney <bchaney@akamai.com>
>
> I don't get why is this a device property?
> It's clearly a backend thing?
>
Hmm.
We want to be able to disable fd-migration per device, when common "local"
migration parameter is on.
That's why we have parameter for virtio-net. It's also good, that we may
enable it by default for newer machine types, to make fd-migration a default
path.
To be honest, it seems that the only thing in this patch (except for interface),
which is not about backand, is that we want to do virtio_net_update_host_features()
after backends incoming migration finished. For this, it's comfortable to have
backend migration as part of device migration stream.
--
Imagine, we move "local-migration" parameter to TAP device. This raise a lot of questions:
1. We loss a possibility to set good default in machine type. Ok, we probably may add
a property to "migration" object, but this property will look like "TAP-local-migration",
and raise discussion again, that it should be property of TAP..
2. We loss possiblity to use qom-set, to change the property value, as net backends are
not Qobjects.. So it will need additional tap-change qmp command or something like this.
3. Migration it self: should we register specific migration state for TAP backend (which is
not very comfortable, because net backends are not devices), or just keep it as part of
virtio-net migration state (like in this patch, but instead of checking for own local-migration
property, check for all backends local-migration properties).
If create personal migration state for TAP backend, detached from virtio-net migration state,
we'll need somehow call virtio_net_update_host_features() _after_ all backends migrated. May
be we can do it in device start, but it's not good, as at this point any error is more critical
than during migration process.. Another option is migration state priorities, but I didn't dig
into it, and it doesn't seem as reliable as simply migrate backends as part of parent device
migration state.
--
What do you think? Theoretically, I can move "local-migration" property to TAP backend, but
seems, it will lead to more problems.
>
>
>> ---
>> hw/core/machine.c | 1 +
>> hw/i386/pc_q35.c | 1 +
>> hw/net/virtio-net.c | 137 ++++++++++++++++++++++++++++++++-
>> include/hw/virtio/virtio-net.h | 2 +
>> include/net/net.h | 2 +
>> 5 files changed, 142 insertions(+), 1 deletion(-)
>>
>> diff --git a/hw/core/machine.c b/hw/core/machine.c
>> index 63baff859f3..619e80c1cb3 100644
>> --- a/hw/core/machine.c
>> +++ b/hw/core/machine.c
>> @@ -41,6 +41,7 @@
>>
>> GlobalProperty hw_compat_11_0[] = {
>> { "chardev-vc", "encoding", "cp437" },
>> + { TYPE_VIRTIO_NET, "local-migration", "false" },
>> };
>> const size_t hw_compat_11_0_len = G_N_ELEMENTS(hw_compat_11_0);
>>
>> diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
>> index d8fed698c72..b5c0e302d59 100644
>> --- a/hw/i386/pc_q35.c
>> +++ b/hw/i386/pc_q35.c
>> @@ -368,6 +368,7 @@ static void pc_q35_machine_options(MachineClass *m)
>> static void pc_q35_machine_11_1_options(MachineClass *m)
>> {
>> pc_q35_machine_options(m);
>> + compat_props_add(m->compat_props, hw_compat_11_0, hw_compat_11_0_len);
>> }
>>
>> DEFINE_Q35_MACHINE_AS_LATEST(11, 1);
>> diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
>> index 2a5d642a647..158b9247a58 100644
>> --- a/hw/net/virtio-net.c
>> +++ b/hw/net/virtio-net.c
>> @@ -38,8 +38,10 @@
>> #include "qapi/qapi-events-migration.h"
>> #include "hw/virtio/virtio-access.h"
>> #include "migration/misc.h"
>> +#include "migration/options.h"
>> #include "standard-headers/linux/ethtool.h"
>> #include "system/system.h"
>> +#include "system/runstate.h"
>> #include "system/replay.h"
>> #include "trace.h"
>> #include "monitor/qdev.h"
>> @@ -3060,7 +3062,17 @@ static void virtio_net_set_multiqueue(VirtIONet *n, int multiqueue)
>> n->multiqueue = multiqueue;
>> virtio_net_change_num_queues(n, max * 2 + 1);
>>
>> - virtio_net_set_queue_pairs(n);
>> + /*
>> + * virtio_net_set_multiqueue() called from set_features(0) on early
>> + * reset, when peer may wait for incoming (and is not initialized
>> + * yet).
>> + * Don't worry about it: virtio_net_set_queue_pairs() will be called
>> + * later form virtio_net_post_load_device(), and anyway will be
>> + * noop for local incoming migration with live backend passing.
>> + */
>> + if (!n->peers_wait_incoming) {
>> + virtio_net_set_queue_pairs(n);
>> + }
>> }
>>
>> static int virtio_net_pre_load_queues(VirtIODevice *vdev, uint32_t n)
>> @@ -3089,6 +3101,17 @@ static void virtio_net_get_features(VirtIODevice *vdev, uint64_t *features,
>>
>> virtio_add_feature_ex(features, VIRTIO_NET_F_MAC);
>>
>> + if (n->peers_wait_incoming) {
>> + /*
>> + * Excessive feature set is OK for early initialization when
>> + * we wait for local incoming migration: actual guest-negotiated
>> + * features will come with migration stream anyway. And we are sure
>> + * that we support same host-features as source, because the backend
>> + * is the same (the same TAP device, for example).
>> + */
>> + return;
>> + }
>> +
>> if (!peer_has_vnet_hdr(n)) {
>> virtio_clear_feature_ex(features, VIRTIO_NET_F_CSUM);
>> virtio_clear_feature_ex(features, VIRTIO_NET_F_HOST_TSO4);
>> @@ -3179,6 +3202,18 @@ static void virtio_net_get_features(VirtIODevice *vdev, uint64_t *features,
>> }
>> }
>>
>> +static bool virtio_net_update_host_features(VirtIONet *n, Error **errp)
>> +{
>> + ERRP_GUARD();
>> + VirtIODevice *vdev = VIRTIO_DEVICE(n);
>> +
>> + peer_test_vnet_hdr(n);
>> +
>> + virtio_net_get_features(vdev, &vdev->host_features, errp);
>> +
>> + return !*errp;
>> +}
>> +
>> static int virtio_net_post_load_device(void *opaque, int version_id)
>> {
>> VirtIONet *n = opaque;
>> @@ -3300,6 +3335,9 @@ struct VirtIONetMigTmp {
>> uint16_t curr_queue_pairs_1;
>> uint8_t has_ufo;
>> uint32_t has_vnet_hdr;
>> +
>> + NetClientState *ncs;
>> + uint32_t max_queue_pairs;
>> };
>>
>> /* The 2nd and subsequent tx_waiting flags are loaded later than
>> @@ -3569,6 +3607,57 @@ static const VMStateDescription vhost_user_net_backend_state = {
>> }
>> };
>>
>> +static bool virtio_net_migrate_local(void *opaque, int version_id)
>> +{
>> + VirtIONet *n = opaque;
>> +
>> + return migrate_local() && n->local_migration;
>> +}
>> +
>> +static int virtio_net_nic_pre_save(void *opaque)
>> +{
>> + struct VirtIONetMigTmp *tmp = opaque;
>> +
>> + tmp->ncs = tmp->parent->nic->ncs;
>> + tmp->max_queue_pairs = tmp->parent->max_queue_pairs;
>> +
>> + return 0;
>> +}
>> +
>> +static int virtio_net_nic_pre_load(void *opaque)
>> +{
>> + /* Reuse the pointer setup from save */
>> + virtio_net_nic_pre_save(opaque);
>> +
>> + return 0;
>> +}
>> +
>> +static int virtio_net_nic_post_load(void *opaque, int version_id)
>> +{
>> + struct VirtIONetMigTmp *tmp = opaque;
>> + Error *local_err = NULL;
>> +
>> + if (!virtio_net_update_host_features(tmp->parent, &local_err)) {
>> + error_report_err(local_err);
>> + return -EINVAL;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static const VMStateDescription vmstate_virtio_net_nic = {
>> + .name = "virtio-net-nic",
>> + .pre_load = virtio_net_nic_pre_load,
>> + .pre_save = virtio_net_nic_pre_save,
>> + .post_load = virtio_net_nic_post_load,
>> + .fields = (const VMStateField[]) {
>> + VMSTATE_VARRAY_UINT32(ncs, struct VirtIONetMigTmp,
>> + max_queue_pairs, 0, vmstate_net_peer_backend,
>> + NetClientState),
>> + VMSTATE_END_OF_LIST()
>> + },
>> +};
>> +
>> static const VMStateDescription vmstate_virtio_net_device = {
>> .name = "virtio-net-device",
>> .version_id = VIRTIO_NET_VM_VERSION,
>> @@ -3600,6 +3689,9 @@ static const VMStateDescription vmstate_virtio_net_device = {
>> * but based on the uint.
>> */
>> VMSTATE_BUFFER_POINTER_UNSAFE(vlans, VirtIONet, 0, MAX_VLAN >> 3),
>> + VMSTATE_WITH_TMP_TEST(VirtIONet, virtio_net_migrate_local,
>> + struct VirtIONetMigTmp,
>> + vmstate_virtio_net_nic),
>> VMSTATE_WITH_TMP(VirtIONet, struct VirtIONetMigTmp,
>> vmstate_virtio_net_has_vnet),
>> VMSTATE_UINT8(mac_table.multi_overflow, VirtIONet),
>> @@ -3864,6 +3956,42 @@ static bool failover_hide_primary_device(DeviceListener *listener,
>> return qatomic_read(&n->failover_primary_hidden);
>> }
>>
>> +static bool virtio_net_check_peers_wait_incoming(VirtIONet *n, bool *waiting,
>> + Error **errp)
>> +{
>> + bool has_waiting = false;
>> + bool has_not_waiting = false;
>> +
>> + for (int i = 0; i < n->max_queue_pairs; i++) {
>> + NetClientState *peer = n->nic->ncs[i].peer;
>> + if (!peer) {
>> + continue;
>> + }
>> +
>> + if (peer->info->is_wait_incoming &&
>> + peer->info->is_wait_incoming(peer)) {
>> + has_waiting = true;
>> + } else {
>> + has_not_waiting = true;
>> + }
>> +
>> + if (has_waiting && has_not_waiting) {
>> + error_setg(errp, "Mixed peer states: some peers wait for incoming "
>> + "migration while others don't");
>> + return false;
>> + }
>> + }
>> +
>> + if (has_waiting && !runstate_check(RUN_STATE_INMIGRATE)) {
>> + error_setg(errp, "Peers wait for incoming, but it's not an incoming "
>> + "migration.");
>> + return false;
>> + }
>> +
>> + *waiting = has_waiting;
>> + return true;
>> +}
>> +
>> static void virtio_net_device_realize(DeviceState *dev, Error **errp)
>> {
>> VirtIODevice *vdev = VIRTIO_DEVICE(dev);
>> @@ -4001,6 +4129,12 @@ static void virtio_net_device_realize(DeviceState *dev, Error **errp)
>> n->nic->ncs[i].do_not_pad = true;
>> }
>>
>> + if (!virtio_net_check_peers_wait_incoming(n, &n->peers_wait_incoming,
>> + errp)) {
>> + virtio_cleanup(vdev);
>> + return;
>> + }
>> +
>> peer_test_vnet_hdr(n);
>> if (peer_has_vnet_hdr(n)) {
>> n->host_hdr_len = sizeof(struct virtio_net_hdr);
>> @@ -4310,6 +4444,7 @@ static const Property virtio_net_properties[] = {
>> host_features_ex,
>> VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO_CSUM,
>> true),
>> + DEFINE_PROP_BOOL("local-migration", VirtIONet, local_migration, true),
>> };
>>
>> static void virtio_net_class_init(ObjectClass *klass, const void *data)
>> diff --git a/include/hw/virtio/virtio-net.h b/include/hw/virtio/virtio-net.h
>> index 371e3764282..0c14e314409 100644
>> --- a/include/hw/virtio/virtio-net.h
>> +++ b/include/hw/virtio/virtio-net.h
>> @@ -230,6 +230,8 @@ struct VirtIONet {
>> struct EBPFRSSContext ebpf_rss;
>> uint32_t nr_ebpf_rss_fds;
>> char **ebpf_rss_fds;
>> + bool peers_wait_incoming;
>> + bool local_migration;
>> };
>>
>> size_t virtio_net_handle_ctrl_iov(VirtIODevice *vdev,
>> diff --git a/include/net/net.h b/include/net/net.h
>> index aa34043b1ac..d4cf399d4a8 100644
>> --- a/include/net/net.h
>> +++ b/include/net/net.h
>> @@ -82,6 +82,7 @@ typedef void (SocketReadStateFinalize)(SocketReadState *rs);
>> typedef void (NetAnnounce)(NetClientState *);
>> typedef bool (SetSteeringEBPF)(NetClientState *, int);
>> typedef bool (NetCheckPeerType)(NetClientState *, ObjectClass *, Error **);
>> +typedef bool (IsWaitIncoming)(NetClientState *);
>> typedef struct vhost_net *(GetVHostNet)(NetClientState *nc);
>>
>> typedef struct NetClientInfo {
>> @@ -110,6 +111,7 @@ typedef struct NetClientInfo {
>> NetAnnounce *announce;
>> SetSteeringEBPF *set_steering_ebpf;
>> NetCheckPeerType *check_peer_type;
>> + IsWaitIncoming *is_wait_incoming;
>> GetVHostNet *get_vhost_net;
>> const VMStateDescription *backend_vmsd;
>> } NetClientInfo;
>> --
>> 2.52.0
>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v16 5/8] virtio-net: support local migration of backend
2026-05-25 13:51 ` Vladimir Sementsov-Ogievskiy
@ 2026-05-25 14:45 ` Peter Xu
2026-05-26 11:23 ` Vladimir Sementsov-Ogievskiy
0 siblings, 1 reply; 25+ messages in thread
From: Peter Xu @ 2026-05-25 14:45 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy
Cc: Michael S. Tsirkin, jasowang, armbru, farosas, raphael.s.norwitz,
bchaney, qemu-devel, berrange, pbonzini, yc-core,
Philippe Mathieu-Daudé, Zhao Liu, Richard Henderson
On Mon, May 25, 2026 at 04:51:55PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> On 24.05.26 12:09, Michael S. Tsirkin wrote:
> > On Fri, May 22, 2026 at 03:05:30PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> > > Add virtio-net option local-migration, which is true by default,
> > > but false for older machine types, which doesn't support the feature.
> > >
> > > When both global migration parameter "local" and new virtio-net
> > > parameter "local-migration" are true, virtio-net transfer the whole
> > > net backend to the destination, including open file descriptors.
> > > Of-course, its only for local migration and the channel must be
> > > UNIX domain socket.
> > >
> > > This way management tool should not care about creating new TAP, and
> > > should not handle switching to it. Migration downtime become shorter.
> > >
> > > Support for TAP will come in the next commit.
> > >
> > > Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
> > > Reviewed-by: Ben Chaney <bchaney@akamai.com>
> >
> > I don't get why is this a device property?
> > It's clearly a backend thing?
> >
>
> Hmm.
>
> We want to be able to disable fd-migration per device, when common "local"
> migration parameter is on.
>
> That's why we have parameter for virtio-net. It's also good, that we may
> enable it by default for newer machine types, to make fd-migration a default
> path.
>
> To be honest, it seems that the only thing in this patch (except for interface),
> which is not about backand, is that we want to do virtio_net_update_host_features()
> after backends incoming migration finished. For this, it's comfortable to have
> backend migration as part of device migration stream.
>
> --
>
> Imagine, we move "local-migration" parameter to TAP device. This raise a lot of questions:
Hmm, I thought we discussed this quite some time ago.. I can't remember
details, but I'll try to comment with what I can still remember.
>
> 1. We loss a possibility to set good default in machine type. Ok, we probably may add
> a property to "migration" object, but this property will look like "TAP-local-migration",
> and raise discussion again, that it should be property of TAP..
If you recall I worked on the other series because of this desire of having
tap be able to be QOMified and also support machine compat properties:
https://lore.kernel.org/all/20251209162857.857593-2-peterx@redhat.com/
I didn't get a lot of feedback supporting having TYPE_OBJECT_COMPAT, maybe
it's an overkill only to apply object_apply_compat_props().
However, just to say we can still QOMify TAP and then add its own
instance_post_init() to also do object_apply_compat_props(), like quite a
few other existing users, then this (1) isn't a problem, and it doesn't
need to depend on my series either.
>
> 2. We loss possiblity to use qom-set, to change the property value, as net backends are
> not Qobjects.. So it will need additional tap-change qmp command or something like this.
This also shouldn't be a problem if we QOMify TAP, IIUC.
>
> 3. Migration it self: should we register specific migration state for TAP backend (which is
> not very comfortable, because net backends are not devices), or just keep it as part of
> virtio-net migration state (like in this patch, but instead of checking for own local-migration
> property, check for all backends local-migration properties).
>
> If create personal migration state for TAP backend, detached from virtio-net migration state,
> we'll need somehow call virtio_net_update_host_features() _after_ all backends migrated. May
> be we can do it in device start, but it's not good, as at this point any error is more critical
> than during migration process.. Another option is migration state priorities, but I didn't dig
> into it, and it doesn't seem as reliable as simply migrate backends as part of parent device
> migration state.
If this is about the "which VMSD to migrate earlier" problem, I recall we
discussed about using MigrationPriority to serialize them. But I'm not
sure if this is exactly the same issue.
Thanks,
>
> --
>
> What do you think? Theoretically, I can move "local-migration" property to TAP backend, but
> seems, it will lead to more problems.
>
> >
> >
> > > ---
> > > hw/core/machine.c | 1 +
> > > hw/i386/pc_q35.c | 1 +
> > > hw/net/virtio-net.c | 137 ++++++++++++++++++++++++++++++++-
> > > include/hw/virtio/virtio-net.h | 2 +
> > > include/net/net.h | 2 +
> > > 5 files changed, 142 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/hw/core/machine.c b/hw/core/machine.c
> > > index 63baff859f3..619e80c1cb3 100644
> > > --- a/hw/core/machine.c
> > > +++ b/hw/core/machine.c
> > > @@ -41,6 +41,7 @@
> > > GlobalProperty hw_compat_11_0[] = {
> > > { "chardev-vc", "encoding", "cp437" },
> > > + { TYPE_VIRTIO_NET, "local-migration", "false" },
> > > };
> > > const size_t hw_compat_11_0_len = G_N_ELEMENTS(hw_compat_11_0);
> > > diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
> > > index d8fed698c72..b5c0e302d59 100644
> > > --- a/hw/i386/pc_q35.c
> > > +++ b/hw/i386/pc_q35.c
> > > @@ -368,6 +368,7 @@ static void pc_q35_machine_options(MachineClass *m)
> > > static void pc_q35_machine_11_1_options(MachineClass *m)
> > > {
> > > pc_q35_machine_options(m);
> > > + compat_props_add(m->compat_props, hw_compat_11_0, hw_compat_11_0_len);
> > > }
> > > DEFINE_Q35_MACHINE_AS_LATEST(11, 1);
> > > diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
> > > index 2a5d642a647..158b9247a58 100644
> > > --- a/hw/net/virtio-net.c
> > > +++ b/hw/net/virtio-net.c
> > > @@ -38,8 +38,10 @@
> > > #include "qapi/qapi-events-migration.h"
> > > #include "hw/virtio/virtio-access.h"
> > > #include "migration/misc.h"
> > > +#include "migration/options.h"
> > > #include "standard-headers/linux/ethtool.h"
> > > #include "system/system.h"
> > > +#include "system/runstate.h"
> > > #include "system/replay.h"
> > > #include "trace.h"
> > > #include "monitor/qdev.h"
> > > @@ -3060,7 +3062,17 @@ static void virtio_net_set_multiqueue(VirtIONet *n, int multiqueue)
> > > n->multiqueue = multiqueue;
> > > virtio_net_change_num_queues(n, max * 2 + 1);
> > > - virtio_net_set_queue_pairs(n);
> > > + /*
> > > + * virtio_net_set_multiqueue() called from set_features(0) on early
> > > + * reset, when peer may wait for incoming (and is not initialized
> > > + * yet).
> > > + * Don't worry about it: virtio_net_set_queue_pairs() will be called
> > > + * later form virtio_net_post_load_device(), and anyway will be
> > > + * noop for local incoming migration with live backend passing.
> > > + */
> > > + if (!n->peers_wait_incoming) {
> > > + virtio_net_set_queue_pairs(n);
> > > + }
> > > }
> > > static int virtio_net_pre_load_queues(VirtIODevice *vdev, uint32_t n)
> > > @@ -3089,6 +3101,17 @@ static void virtio_net_get_features(VirtIODevice *vdev, uint64_t *features,
> > > virtio_add_feature_ex(features, VIRTIO_NET_F_MAC);
> > > + if (n->peers_wait_incoming) {
> > > + /*
> > > + * Excessive feature set is OK for early initialization when
> > > + * we wait for local incoming migration: actual guest-negotiated
> > > + * features will come with migration stream anyway. And we are sure
> > > + * that we support same host-features as source, because the backend
> > > + * is the same (the same TAP device, for example).
> > > + */
> > > + return;
> > > + }
> > > +
> > > if (!peer_has_vnet_hdr(n)) {
> > > virtio_clear_feature_ex(features, VIRTIO_NET_F_CSUM);
> > > virtio_clear_feature_ex(features, VIRTIO_NET_F_HOST_TSO4);
> > > @@ -3179,6 +3202,18 @@ static void virtio_net_get_features(VirtIODevice *vdev, uint64_t *features,
> > > }
> > > }
> > > +static bool virtio_net_update_host_features(VirtIONet *n, Error **errp)
> > > +{
> > > + ERRP_GUARD();
> > > + VirtIODevice *vdev = VIRTIO_DEVICE(n);
> > > +
> > > + peer_test_vnet_hdr(n);
> > > +
> > > + virtio_net_get_features(vdev, &vdev->host_features, errp);
> > > +
> > > + return !*errp;
> > > +}
> > > +
> > > static int virtio_net_post_load_device(void *opaque, int version_id)
> > > {
> > > VirtIONet *n = opaque;
> > > @@ -3300,6 +3335,9 @@ struct VirtIONetMigTmp {
> > > uint16_t curr_queue_pairs_1;
> > > uint8_t has_ufo;
> > > uint32_t has_vnet_hdr;
> > > +
> > > + NetClientState *ncs;
> > > + uint32_t max_queue_pairs;
> > > };
> > > /* The 2nd and subsequent tx_waiting flags are loaded later than
> > > @@ -3569,6 +3607,57 @@ static const VMStateDescription vhost_user_net_backend_state = {
> > > }
> > > };
> > > +static bool virtio_net_migrate_local(void *opaque, int version_id)
> > > +{
> > > + VirtIONet *n = opaque;
> > > +
> > > + return migrate_local() && n->local_migration;
> > > +}
> > > +
> > > +static int virtio_net_nic_pre_save(void *opaque)
> > > +{
> > > + struct VirtIONetMigTmp *tmp = opaque;
> > > +
> > > + tmp->ncs = tmp->parent->nic->ncs;
> > > + tmp->max_queue_pairs = tmp->parent->max_queue_pairs;
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static int virtio_net_nic_pre_load(void *opaque)
> > > +{
> > > + /* Reuse the pointer setup from save */
> > > + virtio_net_nic_pre_save(opaque);
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static int virtio_net_nic_post_load(void *opaque, int version_id)
> > > +{
> > > + struct VirtIONetMigTmp *tmp = opaque;
> > > + Error *local_err = NULL;
> > > +
> > > + if (!virtio_net_update_host_features(tmp->parent, &local_err)) {
> > > + error_report_err(local_err);
> > > + return -EINVAL;
> > > + }
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static const VMStateDescription vmstate_virtio_net_nic = {
> > > + .name = "virtio-net-nic",
> > > + .pre_load = virtio_net_nic_pre_load,
> > > + .pre_save = virtio_net_nic_pre_save,
> > > + .post_load = virtio_net_nic_post_load,
> > > + .fields = (const VMStateField[]) {
> > > + VMSTATE_VARRAY_UINT32(ncs, struct VirtIONetMigTmp,
> > > + max_queue_pairs, 0, vmstate_net_peer_backend,
> > > + NetClientState),
> > > + VMSTATE_END_OF_LIST()
> > > + },
> > > +};
> > > +
> > > static const VMStateDescription vmstate_virtio_net_device = {
> > > .name = "virtio-net-device",
> > > .version_id = VIRTIO_NET_VM_VERSION,
> > > @@ -3600,6 +3689,9 @@ static const VMStateDescription vmstate_virtio_net_device = {
> > > * but based on the uint.
> > > */
> > > VMSTATE_BUFFER_POINTER_UNSAFE(vlans, VirtIONet, 0, MAX_VLAN >> 3),
> > > + VMSTATE_WITH_TMP_TEST(VirtIONet, virtio_net_migrate_local,
> > > + struct VirtIONetMigTmp,
> > > + vmstate_virtio_net_nic),
> > > VMSTATE_WITH_TMP(VirtIONet, struct VirtIONetMigTmp,
> > > vmstate_virtio_net_has_vnet),
> > > VMSTATE_UINT8(mac_table.multi_overflow, VirtIONet),
> > > @@ -3864,6 +3956,42 @@ static bool failover_hide_primary_device(DeviceListener *listener,
> > > return qatomic_read(&n->failover_primary_hidden);
> > > }
> > > +static bool virtio_net_check_peers_wait_incoming(VirtIONet *n, bool *waiting,
> > > + Error **errp)
> > > +{
> > > + bool has_waiting = false;
> > > + bool has_not_waiting = false;
> > > +
> > > + for (int i = 0; i < n->max_queue_pairs; i++) {
> > > + NetClientState *peer = n->nic->ncs[i].peer;
> > > + if (!peer) {
> > > + continue;
> > > + }
> > > +
> > > + if (peer->info->is_wait_incoming &&
> > > + peer->info->is_wait_incoming(peer)) {
> > > + has_waiting = true;
> > > + } else {
> > > + has_not_waiting = true;
> > > + }
> > > +
> > > + if (has_waiting && has_not_waiting) {
> > > + error_setg(errp, "Mixed peer states: some peers wait for incoming "
> > > + "migration while others don't");
> > > + return false;
> > > + }
> > > + }
> > > +
> > > + if (has_waiting && !runstate_check(RUN_STATE_INMIGRATE)) {
> > > + error_setg(errp, "Peers wait for incoming, but it's not an incoming "
> > > + "migration.");
> > > + return false;
> > > + }
> > > +
> > > + *waiting = has_waiting;
> > > + return true;
> > > +}
> > > +
> > > static void virtio_net_device_realize(DeviceState *dev, Error **errp)
> > > {
> > > VirtIODevice *vdev = VIRTIO_DEVICE(dev);
> > > @@ -4001,6 +4129,12 @@ static void virtio_net_device_realize(DeviceState *dev, Error **errp)
> > > n->nic->ncs[i].do_not_pad = true;
> > > }
> > > + if (!virtio_net_check_peers_wait_incoming(n, &n->peers_wait_incoming,
> > > + errp)) {
> > > + virtio_cleanup(vdev);
> > > + return;
> > > + }
> > > +
> > > peer_test_vnet_hdr(n);
> > > if (peer_has_vnet_hdr(n)) {
> > > n->host_hdr_len = sizeof(struct virtio_net_hdr);
> > > @@ -4310,6 +4444,7 @@ static const Property virtio_net_properties[] = {
> > > host_features_ex,
> > > VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO_CSUM,
> > > true),
> > > + DEFINE_PROP_BOOL("local-migration", VirtIONet, local_migration, true),
> > > };
> > > static void virtio_net_class_init(ObjectClass *klass, const void *data)
> > > diff --git a/include/hw/virtio/virtio-net.h b/include/hw/virtio/virtio-net.h
> > > index 371e3764282..0c14e314409 100644
> > > --- a/include/hw/virtio/virtio-net.h
> > > +++ b/include/hw/virtio/virtio-net.h
> > > @@ -230,6 +230,8 @@ struct VirtIONet {
> > > struct EBPFRSSContext ebpf_rss;
> > > uint32_t nr_ebpf_rss_fds;
> > > char **ebpf_rss_fds;
> > > + bool peers_wait_incoming;
> > > + bool local_migration;
> > > };
> > > size_t virtio_net_handle_ctrl_iov(VirtIODevice *vdev,
> > > diff --git a/include/net/net.h b/include/net/net.h
> > > index aa34043b1ac..d4cf399d4a8 100644
> > > --- a/include/net/net.h
> > > +++ b/include/net/net.h
> > > @@ -82,6 +82,7 @@ typedef void (SocketReadStateFinalize)(SocketReadState *rs);
> > > typedef void (NetAnnounce)(NetClientState *);
> > > typedef bool (SetSteeringEBPF)(NetClientState *, int);
> > > typedef bool (NetCheckPeerType)(NetClientState *, ObjectClass *, Error **);
> > > +typedef bool (IsWaitIncoming)(NetClientState *);
> > > typedef struct vhost_net *(GetVHostNet)(NetClientState *nc);
> > > typedef struct NetClientInfo {
> > > @@ -110,6 +111,7 @@ typedef struct NetClientInfo {
> > > NetAnnounce *announce;
> > > SetSteeringEBPF *set_steering_ebpf;
> > > NetCheckPeerType *check_peer_type;
> > > + IsWaitIncoming *is_wait_incoming;
> > > GetVHostNet *get_vhost_net;
> > > const VMStateDescription *backend_vmsd;
> > > } NetClientInfo;
> > > --
> > > 2.52.0
> >
>
>
> --
> Best regards,
> Vladimir
>
--
Peter Xu
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v16 5/8] virtio-net: support local migration of backend
2026-05-25 14:45 ` Peter Xu
@ 2026-05-26 11:23 ` Vladimir Sementsov-Ogievskiy
2026-05-26 17:47 ` Peter Xu
2026-05-27 11:35 ` Vladimir Sementsov-Ogievskiy
0 siblings, 2 replies; 25+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2026-05-26 11:23 UTC (permalink / raw)
To: Peter Xu
Cc: Michael S. Tsirkin, jasowang, armbru, farosas, raphael.s.norwitz,
bchaney, qemu-devel, berrange, pbonzini, yc-core,
Philippe Mathieu-Daudé, Zhao Liu, Richard Henderson
On 25.05.26 17:45, Peter Xu wrote:
> On Mon, May 25, 2026 at 04:51:55PM +0300, Vladimir Sementsov-Ogievskiy wrote:
>> On 24.05.26 12:09, Michael S. Tsirkin wrote:
>>> On Fri, May 22, 2026 at 03:05:30PM +0300, Vladimir Sementsov-Ogievskiy wrote:
>>>> Add virtio-net option local-migration, which is true by default,
>>>> but false for older machine types, which doesn't support the feature.
>>>>
>>>> When both global migration parameter "local" and new virtio-net
>>>> parameter "local-migration" are true, virtio-net transfer the whole
>>>> net backend to the destination, including open file descriptors.
>>>> Of-course, its only for local migration and the channel must be
>>>> UNIX domain socket.
>>>>
>>>> This way management tool should not care about creating new TAP, and
>>>> should not handle switching to it. Migration downtime become shorter.
>>>>
>>>> Support for TAP will come in the next commit.
>>>>
>>>> Signed-off-by: Vladimir Sementsov-Ogievskiy<vsementsov@yandex-team.ru>
>>>> Reviewed-by: Ben Chaney<bchaney@akamai.com>
>>> I don't get why is this a device property?
>>> It's clearly a backend thing?
>>>
>> Hmm.
>>
>> We want to be able to disable fd-migration per device, when common "local"
>> migration parameter is on.
>>
>> That's why we have parameter for virtio-net. It's also good, that we may
>> enable it by default for newer machine types, to make fd-migration a default
>> path.
>>
>> To be honest, it seems that the only thing in this patch (except for interface),
>> which is not about backand, is that we want to do virtio_net_update_host_features()
>> after backends incoming migration finished. For this, it's comfortable to have
>> backend migration as part of device migration stream.
>>
>> --
>>
>> Imagine, we move "local-migration" parameter to TAP device. This raise a lot of questions:
> Hmm, I thought we discussed this quite some time ago.. I can't remember
> details, but I'll try to comment with what I can still remember.
>
>> 1. We loss a possibility to set good default in machine type. Ok, we probably may add
>> a property to "migration" object, but this property will look like "TAP-local-migration",
>> and raise discussion again, that it should be property of TAP..
> If you recall I worked on the other series because of this desire of having
> tap be able to be QOMified and also support machine compat properties:
>
> https://lore.kernel.org/all/20251209162857.857593-2-peterx@redhat.com/
>
> I didn't get a lot of feedback supporting having TYPE_OBJECT_COMPAT, maybe
> it's an overkill only to apply object_apply_compat_props().
>
> However, just to say we can still QOMify TAP and then add its own
> instance_post_init() to also do object_apply_compat_props(), like quite a
> few other existing users, then this (1) isn't a problem, and it doesn't
> need to depend on my series either.
Thanks for reminding and explanation. Previously, I thought that
QOMification is huge task, and your series is required. Now I see,
that it is not so difficult. All that's left for me to say is
"let's try it")
Here is an experimental diff on top on this series. If you like it,
I'll resend with proper division into commits:
diff --git a/hw/core/machine.c b/hw/core/machine.c
index 619e80c1cb3..67e873c74fe 100644
--- a/hw/core/machine.c
+++ b/hw/core/machine.c
@@ -36,12 +36,13 @@
#include "hw/virtio/virtio-pci.h"
#include "hw/virtio/virtio-net.h"
#include "hw/virtio/virtio-iommu.h"
+#include "net/tap.h"
#include "hw/acpi/generic_event_device.h"
#include "qemu/audio.h"
GlobalProperty hw_compat_11_0[] = {
{ "chardev-vc", "encoding", "cp437" },
- { TYPE_VIRTIO_NET, "local-migration", "false" },
+ { TYPE_TAP_NETDEV, "local-migration", "false" },
};
const size_t hw_compat_11_0_len = G_N_ELEMENTS(hw_compat_11_0);
diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index 158b9247a58..bad349d3e83 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -3202,24 +3202,13 @@ static void virtio_net_get_features(VirtIODevice *vdev, uint64_t *features,
}
}
-static bool virtio_net_update_host_features(VirtIONet *n, Error **errp)
-{
- ERRP_GUARD();
- VirtIODevice *vdev = VIRTIO_DEVICE(n);
-
- peer_test_vnet_hdr(n);
-
- virtio_net_get_features(vdev, &vdev->host_features, errp);
-
- return !*errp;
-}
-
static int virtio_net_post_load_device(void *opaque, int version_id)
{
VirtIONet *n = opaque;
VirtIODevice *vdev = VIRTIO_DEVICE(n);
int i, link_down;
bool has_tunnel_hdr = virtio_has_tunnel_hdr(vdev->guest_features_ex);
+ Error *local_err = NULL;
trace_virtio_net_post_load_device();
virtio_net_set_mrg_rx_bufs(n, n->mergeable_rx_bufs,
@@ -3277,6 +3266,19 @@ static int virtio_net_post_load_device(void *opaque, int version_id)
}
virtio_net_commit_rss_config(n);
+
+ /*
+ * The TAP backend has already been migrated at higher priority
+ * (MIG_PRI_TAP) and virtio_net_vnet_post_load() has already called
+ * peer_test_vnet_hdr(). Recompute host_features so that virtio-net
+ * reflects the capabilities of the restored backend.
+ */
+ virtio_net_get_features(vdev, &vdev->host_features, &local_err);
+ if (local_err) {
+ error_report_err(local_err);
+ return -EINVAL;
+ }
+
return 0;
}
@@ -3430,6 +3432,13 @@ static int virtio_net_vnet_post_load(void *opaque, int version_id)
{
struct VirtIONetMigTmp *tmp = opaque;
+ /*
+ * The TAP backend has already been migrated at higher priority
+ * (MIG_PRI_TAP), so n->has_vnet_hdr can be refreshed from the live
+ * backend right here.
+ */
+ peer_test_vnet_hdr(tmp->parent);
+
if (tmp->has_vnet_hdr && !peer_has_vnet_hdr(tmp->parent)) {
error_report("virtio-net: saved image requires vnet_hdr=on");
return -EINVAL;
@@ -3607,57 +3616,6 @@ static const VMStateDescription vhost_user_net_backend_state = {
}
};
-static bool virtio_net_migrate_local(void *opaque, int version_id)
-{
- VirtIONet *n = opaque;
-
- return migrate_local() && n->local_migration;
-}
-
-static int virtio_net_nic_pre_save(void *opaque)
-{
- struct VirtIONetMigTmp *tmp = opaque;
-
- tmp->ncs = tmp->parent->nic->ncs;
- tmp->max_queue_pairs = tmp->parent->max_queue_pairs;
-
- return 0;
-}
-
-static int virtio_net_nic_pre_load(void *opaque)
-{
- /* Reuse the pointer setup from save */
- virtio_net_nic_pre_save(opaque);
-
- return 0;
-}
-
-static int virtio_net_nic_post_load(void *opaque, int version_id)
-{
- struct VirtIONetMigTmp *tmp = opaque;
- Error *local_err = NULL;
-
- if (!virtio_net_update_host_features(tmp->parent, &local_err)) {
- error_report_err(local_err);
- return -EINVAL;
- }
-
- return 0;
-}
-
-static const VMStateDescription vmstate_virtio_net_nic = {
- .name = "virtio-net-nic",
- .pre_load = virtio_net_nic_pre_load,
- .pre_save = virtio_net_nic_pre_save,
- .post_load = virtio_net_nic_post_load,
- .fields = (const VMStateField[]) {
- VMSTATE_VARRAY_UINT32(ncs, struct VirtIONetMigTmp,
- max_queue_pairs, 0, vmstate_net_peer_backend,
- NetClientState),
- VMSTATE_END_OF_LIST()
- },
-};
-
static const VMStateDescription vmstate_virtio_net_device = {
.name = "virtio-net-device",
.version_id = VIRTIO_NET_VM_VERSION,
@@ -3689,9 +3647,6 @@ static const VMStateDescription vmstate_virtio_net_device = {
* but based on the uint.
*/
VMSTATE_BUFFER_POINTER_UNSAFE(vlans, VirtIONet, 0, MAX_VLAN >> 3),
- VMSTATE_WITH_TMP_TEST(VirtIONet, virtio_net_migrate_local,
- struct VirtIONetMigTmp,
- vmstate_virtio_net_nic),
VMSTATE_WITH_TMP(VirtIONet, struct VirtIONetMigTmp,
vmstate_virtio_net_has_vnet),
VMSTATE_UINT8(mac_table.multi_overflow, VirtIONet),
@@ -4444,7 +4399,6 @@ static const Property virtio_net_properties[] = {
host_features_ex,
VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO_CSUM,
true),
- DEFINE_PROP_BOOL("local-migration", VirtIONet, local_migration, true),
};
static void virtio_net_class_init(ObjectClass *klass, const void *data)
diff --git a/include/hw/virtio/virtio-net.h b/include/hw/virtio/virtio-net.h
index 0c14e314409..8c967760c2a 100644
--- a/include/hw/virtio/virtio-net.h
+++ b/include/hw/virtio/virtio-net.h
@@ -231,7 +231,6 @@ struct VirtIONet {
uint32_t nr_ebpf_rss_fds;
char **ebpf_rss_fds;
bool peers_wait_incoming;
- bool local_migration;
};
size_t virtio_net_handle_ctrl_iov(VirtIODevice *vdev,
diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h
index 0a8a2e85a63..e7b3cd7db01 100644
--- a/include/migration/vmstate.h
+++ b/include/migration/vmstate.h
@@ -178,6 +178,7 @@ typedef enum {
MIG_PRI_LOW, /* Must happen after default */
MIG_PRI_DEFAULT,
+ MIG_PRI_TAP, /* Must happen before virtio-net device */
MIG_PRI_IOMMU, /* Must happen before PCI devices */
MIG_PRI_PCI_BUS, /* Must happen before IOMMU */
MIG_PRI_VIRTIO_MEM, /* Must happen before IOMMU */
diff --git a/include/net/net.h b/include/net/net.h
index d4cf399d4a8..be7ca0ce845 100644
--- a/include/net/net.h
+++ b/include/net/net.h
@@ -113,7 +113,6 @@ typedef struct NetClientInfo {
NetCheckPeerType *check_peer_type;
IsWaitIncoming *is_wait_incoming;
GetVHostNet *get_vhost_net;
- const VMStateDescription *backend_vmsd;
} NetClientInfo;
struct NetClientState {
@@ -164,6 +163,13 @@ char *qemu_mac_strdup_printf(const uint8_t *macaddr);
NetClientState *qemu_find_netdev(const char *id);
int qemu_find_net_clients_except(const char *id, NetClientState **ncs,
NetClientDriver type, int max);
+void qemu_net_client_setup(NetClientState *nc,
+ NetClientInfo *info,
+ NetClientState *peer,
+ const char *model,
+ const char *name,
+ NetClientDestructor *destructor,
+ bool is_datapath);
NetClientState *qemu_new_net_client(NetClientInfo *info,
NetClientState *peer,
const char *model,
@@ -358,6 +364,4 @@ static inline bool net_peer_needs_padding(NetClientState *nc)
return nc->peer && !nc->peer->do_not_pad;
}
-extern const VMStateInfo vmstate_net_peer_backend;
-
#endif
diff --git a/include/net/tap.h b/include/net/tap.h
index 6f34f13eae4..325de81f03b 100644
--- a/include/net/tap.h
+++ b/include/net/tap.h
@@ -28,9 +28,12 @@
#include "standard-headers/linux/virtio_net.h"
+#define TYPE_TAP_NETDEV "tap-netdev"
+
int tap_enable(NetClientState *nc);
int tap_disable(NetClientState *nc);
int tap_get_fd(NetClientState *nc);
+bool tap_get_local_migration(NetClientState *nc);
#endif /* QEMU_NET_TAP_H */
diff --git a/net/net.c b/net/net.c
index 8bccb1880a1..34766763ae5 100644
--- a/net/net.c
+++ b/net/net.c
@@ -58,7 +58,6 @@
#include "qapi/string-output-visitor.h"
#include "qapi/qobject-input-visitor.h"
#include "standard-headers/linux/virtio_net.h"
-#include "migration/vmstate.h"
/* Net bridge is currently not supported for W32. */
#if !defined(_WIN32)
@@ -262,13 +261,13 @@ static ssize_t qemu_deliver_packet_iov(NetClientState *sender,
int iovcnt,
void *opaque);
-static void qemu_net_client_setup(NetClientState *nc,
- NetClientInfo *info,
- NetClientState *peer,
- const char *model,
- const char *name,
- NetClientDestructor *destructor,
- bool is_datapath)
+void qemu_net_client_setup(NetClientState *nc,
+ NetClientInfo *info,
+ NetClientState *peer,
+ const char *model,
+ const char *name,
+ NetClientDestructor *destructor,
+ bool is_datapath)
{
nc->info = info;
nc->model = g_strdup(model);
@@ -2175,48 +2174,3 @@ int net_fill_rstate(SocketReadState *rs, const uint8_t *buf, int size)
return 0;
}
-static int get_peer_backend(QEMUFile *f, void *pv, size_t size,
- const VMStateField *field)
-{
- NetClientState *nc = pv;
- Error *local_err = NULL;
- int ret;
-
- if (!nc->peer) {
- return -EINVAL;
- }
- nc = nc->peer;
-
- ret = vmstate_load_state(f, nc->info->backend_vmsd, nc, 0, &local_err);
- if (ret < 0) {
- error_report_err(local_err);
- }
-
- return ret;
-}
-
-static int put_peer_backend(QEMUFile *f, void *pv, size_t size,
- const VMStateField *field, JSONWriter *vmdesc)
-{
- NetClientState *nc = pv;
- Error *local_err = NULL;
- int ret;
-
- if (!nc->peer) {
- return -EINVAL;
- }
- nc = nc->peer;
-
- ret = vmstate_save_state(f, nc->info->backend_vmsd, nc, 0, &local_err);
- if (ret < 0) {
- error_report_err(local_err);
- }
-
- return ret;
-}
-
-const VMStateInfo vmstate_net_peer_backend = {
- .name = "virtio-net-nic-nc-backend",
- .get = get_peer_backend,
- .put = put_peer_backend,
-};
diff --git a/net/tap.c b/net/tap.c
index 9b1d4613a0b..e3aeb99d249 100644
--- a/net/tap.c
+++ b/net/tap.c
@@ -38,12 +38,17 @@
#include "monitor/monitor.h"
#include "system/runstate.h"
#include "system/system.h"
+#include "migration/misc.h"
#include "qapi/error.h"
#include "qemu/cutils.h"
#include "qemu/error-report.h"
#include "qemu/main-loop.h"
#include "qemu/sockets.h"
#include "hw/virtio/vhost.h"
+#include "hw/core/vmstate-if.h"
+#include "migration/vmstate.h"
+#include "qom/object.h"
+#include "qom/compat-properties.h"
#include "net/tap.h"
#include "net/util.h"
@@ -69,7 +74,13 @@ static const int kernel_feature_bits[] = {
VHOST_INVALID_FEATURE_BIT
};
-typedef struct TAPState {
+OBJECT_DECLARE_SIMPLE_TYPE(TAPState, TAP_NETDEV)
+
+static const VMStateDescription vmstate_tap;
+
+struct TAPState {
+ Object parent_obj;
+
NetClientState nc;
int fd;
int vhostfd;
@@ -90,7 +101,9 @@ typedef struct TAPState {
bool read_poll_detached;
VMChangeStateEntry *vmstate;
-} TAPState;
+ bool local_migration;
+ int queue_index;
+};
static void launch_script(const char *setup_script, const char *ifname,
int fd, Error **errp);
@@ -182,7 +195,7 @@ static ssize_t tap_write_packet(TAPState *s, const struct iovec *iov, int iovcnt
static ssize_t tap_receive_iov(NetClientState *nc, const struct iovec *iov,
int iovcnt)
{
- TAPState *s = DO_UPCAST(TAPState, nc, nc);
+ TAPState *s = container_of(nc, TAPState, nc);
const struct iovec *iovp = iov;
g_autofree struct iovec *iov_copy = NULL;
struct virtio_net_hdr hdr = { };
@@ -218,7 +231,7 @@ ssize_t tap_read_packet(int tapfd, uint8_t *buf, int maxlen)
static void tap_send_completed(NetClientState *nc, ssize_t len)
{
- TAPState *s = DO_UPCAST(TAPState, nc, nc);
+ TAPState *s = container_of(nc, TAPState, nc);
tap_read_poll(s, true);
}
@@ -278,7 +291,7 @@ static void tap_send(void *opaque)
static bool tap_has_ufo(NetClientState *nc)
{
- TAPState *s = DO_UPCAST(TAPState, nc, nc);
+ TAPState *s = container_of(nc, TAPState, nc);
assert(nc->info->type == NET_CLIENT_DRIVER_TAP);
@@ -287,7 +300,7 @@ static bool tap_has_ufo(NetClientState *nc)
static bool tap_has_uso(NetClientState *nc)
{
- TAPState *s = DO_UPCAST(TAPState, nc, nc);
+ TAPState *s = container_of(nc, TAPState, nc);
assert(nc->info->type == NET_CLIENT_DRIVER_TAP);
@@ -296,7 +309,7 @@ static bool tap_has_uso(NetClientState *nc)
static bool tap_has_tunnel(NetClientState *nc)
{
- TAPState *s = DO_UPCAST(TAPState, nc, nc);
+ TAPState *s = container_of(nc, TAPState, nc);
assert(nc->info->type == NET_CLIENT_DRIVER_TAP);
return s->has_tunnel;
@@ -304,7 +317,7 @@ static bool tap_has_tunnel(NetClientState *nc)
static bool tap_has_vnet_hdr(NetClientState *nc)
{
- TAPState *s = DO_UPCAST(TAPState, nc, nc);
+ TAPState *s = container_of(nc, TAPState, nc);
assert(nc->info->type == NET_CLIENT_DRIVER_TAP);
@@ -318,7 +331,7 @@ static bool tap_has_vnet_hdr_len(NetClientState *nc, int len)
static void tap_set_vnet_hdr_len(NetClientState *nc, int len)
{
- TAPState *s = DO_UPCAST(TAPState, nc, nc);
+ TAPState *s = container_of(nc, TAPState, nc);
assert(nc->info->type == NET_CLIENT_DRIVER_TAP);
@@ -329,21 +342,21 @@ static void tap_set_vnet_hdr_len(NetClientState *nc, int len)
static int tap_set_vnet_le(NetClientState *nc, bool is_le)
{
- TAPState *s = DO_UPCAST(TAPState, nc, nc);
+ TAPState *s = container_of(nc, TAPState, nc);
return tap_fd_set_vnet_le(s->fd, is_le);
}
static int tap_set_vnet_be(NetClientState *nc, bool is_be)
{
- TAPState *s = DO_UPCAST(TAPState, nc, nc);
+ TAPState *s = container_of(nc, TAPState, nc);
return tap_fd_set_vnet_be(s->fd, is_be);
}
static void tap_set_offload(NetClientState *nc, const NetOffloads *ol)
{
- TAPState *s = DO_UPCAST(TAPState, nc, nc);
+ TAPState *s = container_of(nc, TAPState, nc);
if (s->fd < 0) {
return;
}
@@ -364,7 +377,7 @@ static void tap_exit_notify(Notifier *notifier, void *data)
static void tap_cleanup(NetClientState *nc)
{
- TAPState *s = DO_UPCAST(TAPState, nc, nc);
+ TAPState *s = container_of(nc, TAPState, nc);
if (s->vhost_net) {
vhost_net_cleanup(s->vhost_net);
@@ -389,18 +402,20 @@ static void tap_cleanup(NetClientState *nc)
tap_write_poll(s, false);
close(s->fd);
s->fd = -1;
+
+ vmstate_unregister(VMSTATE_IF(s), &vmstate_tap, s);
}
static void tap_poll(NetClientState *nc, bool enable)
{
- TAPState *s = DO_UPCAST(TAPState, nc, nc);
+ TAPState *s = container_of(nc, TAPState, nc);
tap_read_poll(s, enable);
tap_write_poll(s, enable);
}
static bool tap_set_steering_ebpf(NetClientState *nc, int prog_fd)
{
- TAPState *s = DO_UPCAST(TAPState, nc, nc);
+ TAPState *s = container_of(nc, TAPState, nc);
assert(nc->info->type == NET_CLIENT_DRIVER_TAP);
return tap_fd_set_steering_ebpf(s->fd, prog_fd) == 0;
@@ -408,7 +423,7 @@ static bool tap_set_steering_ebpf(NetClientState *nc, int prog_fd)
int tap_get_fd(NetClientState *nc)
{
- TAPState *s = DO_UPCAST(TAPState, nc, nc);
+ TAPState *s = container_of(nc, TAPState, nc);
assert(nc->info->type == NET_CLIENT_DRIVER_TAP);
return s->fd;
}
@@ -420,14 +435,14 @@ int tap_get_fd(NetClientState *nc)
*/
static VHostNetState *tap_get_vhost_net(NetClientState *nc)
{
- TAPState *s = DO_UPCAST(TAPState, nc, nc);
+ TAPState *s = container_of(nc, TAPState, nc);
assert(nc->info->type == NET_CLIENT_DRIVER_TAP);
return s->vhost_net;
}
static bool tap_is_wait_incoming(NetClientState *nc)
{
- TAPState *s = DO_UPCAST(TAPState, nc, nc);
+ TAPState *s = container_of(nc, TAPState, nc);
assert(nc->info->type == NET_CLIENT_DRIVER_TAP);
return s->fd == -1;
}
@@ -468,10 +483,19 @@ static int tap_post_load(void *opaque, int version_id)
return 0;
}
+static bool tap_needed(void *opaque)
+{
+ TAPState *s = opaque;
+
+ return s->local_migration && migrate_local();
+}
+
static const VMStateDescription vmstate_tap = {
.name = "net-tap",
+ .priority = MIG_PRI_TAP,
.pre_load = tap_pre_load,
.post_load = tap_post_load,
+ .needed = tap_needed,
.fields = (const VMStateField[]) {
VMSTATE_FD(fd, TAPState),
VMSTATE_BOOL(using_vnet_hdr, TAPState),
@@ -484,6 +508,63 @@ static const VMStateDescription vmstate_tap = {
}
};
+static char *tap_vmstate_if_get_id(VMStateIf *obj)
+{
+ TAPState *s = TAP_NETDEV(obj);
+ char *res = g_strdup_printf("%s/%d", s->nc.name, s->queue_index);
+ return res;
+}
+
+static bool tap_get_local_migration_prop(Object *obj, Error **errp)
+{
+ TAPState *s = TAP_NETDEV(obj);
+ return s->local_migration;
+}
+
+static void tap_set_local_migration_prop(Object *obj, bool value, Error **errp)
+{
+ TAPState *s = TAP_NETDEV(obj);
+ s->local_migration = value;
+}
+
+static void tap_instance_init(Object *obj)
+{
+ TAPState *s = TAP_NETDEV(obj);
+ s->local_migration = true;
+}
+
+static void tap_class_init(ObjectClass *klass, const void *data)
+{
+ VMStateIfClass *vc = VMSTATE_IF_CLASS(klass);
+
+ vc->get_id = tap_vmstate_if_get_id;
+
+ object_class_property_add_bool(klass, "local-migration",
+ tap_get_local_migration_prop,
+ tap_set_local_migration_prop);
+ object_class_property_set_description(klass, "local-migration",
+ "Migrate the TAP file descriptor to the target (local migration)");
+}
+
+static const TypeInfo tap_netdev_info = {
+ .name = TYPE_TAP_NETDEV,
+ .parent = TYPE_OBJECT,
+ .instance_size = sizeof(TAPState),
+ .instance_init = tap_instance_init,
+ .instance_post_init = object_apply_compat_props,
+ .class_init = tap_class_init,
+ .interfaces = (const InterfaceInfo[]) {
+ { TYPE_VMSTATE_IF },
+ { }
+ },
+};
+
+static void tap_net_client_destructor(NetClientState *nc)
+{
+ TAPState *s = container_of(nc, TAPState, nc);
+ object_unref(OBJECT(s));
+}
+
/* fd support */
static NetClientInfo net_tap_info = {
@@ -505,22 +586,43 @@ static NetClientInfo net_tap_info = {
.set_steering_ebpf = tap_set_steering_ebpf,
.is_wait_incoming = tap_is_wait_incoming,
.get_vhost_net = tap_get_vhost_net,
- .backend_vmsd = &vmstate_tap,
};
+static TAPState *new_tap(NetClientState *peer,
+ const char *model,
+ const char *name,
+ int queue_index,
+ bool has_local_migration,
+ bool local_migration)
+{
+ TAPState *s = TAP_NETDEV(object_new(TYPE_TAP_NETDEV));
+
+ qemu_net_client_setup(&s->nc, &net_tap_info, peer, model, name,
+ tap_net_client_destructor, true);
+
+ s->queue_index = queue_index;
+
+ if (has_local_migration) {
+ s->local_migration = local_migration;
+ }
+
+ vmstate_register(VMSTATE_IF(s), VMSTATE_INSTANCE_ID_ANY, &vmstate_tap, s);
+
+ return s;
+}
+
static TAPState *net_tap_fd_init(NetClientState *peer,
const char *model,
const char *name,
int fd,
- int vnet_hdr)
+ int vnet_hdr,
+ int queue_index,
+ bool has_local_migration,
+ bool local_migration)
{
NetOffloads ol = {};
- NetClientState *nc;
- TAPState *s;
-
- nc = qemu_new_net_client(&net_tap_info, peer, model, name);
-
- s = DO_UPCAST(TAPState, nc, nc);
+ TAPState *s = new_tap(peer, model, name, queue_index,
+ has_local_migration, local_migration);
s->fd = fd;
s->host_vnet_hdr_len = vnet_hdr ? sizeof(struct virtio_net_hdr) : 0;
@@ -757,7 +859,7 @@ int net_init_bridge(const Netdev *netdev, const char *name,
close(fd);
return -1;
}
- s = net_tap_fd_init(peer, "bridge", name, fd, vnet_hdr);
+ s = net_tap_fd_init(peer, "bridge", name, fd, vnet_hdr, 0, true, false);
qemu_set_info_str(&s->nc, "helper=%s,br=%s", helper, br);
@@ -833,10 +935,13 @@ static bool net_init_tap_one(const NetdevTapOptions *tap, NetClientState *peer,
const char *name,
const char *ifname, const char *script,
const char *downscript, int vhostfd,
- int vnet_hdr, int fd, Error **errp)
+ int vnet_hdr, int fd, int queue_index,
+ Error **errp)
{
TAPState *s = net_tap_fd_init(peer, tap->helper ? "bridge" : "tap",
- name, fd, vnet_hdr);
+ name, fd, vnet_hdr, queue_index,
+ tap->has_local_migration,
+ tap->local_migration);
bool sndbuf_required = tap->has_sndbuf;
int sndbuf =
(tap->has_sndbuf && tap->sndbuf) ? MIN(tap->sndbuf, INT_MAX) : INT_MAX;
@@ -1048,13 +1153,11 @@ int net_init_tap(const Netdev *netdev, const char *name,
if (tap->incoming_fds) {
for (i = 0; i < queues; i++) {
- NetClientState *nc;
- TAPState *s;
+ TAPState *s = new_tap(peer, "tap", name, i,
+ tap->has_local_migration, tap->local_migration);
- nc = qemu_new_net_client(&net_tap_info, peer, "tap", name);
- qemu_set_info_str(nc, "incoming");
+ qemu_set_info_str(&s->nc, "incoming");
- s = DO_UPCAST(TAPState, nc, nc);
s->fd = -1;
if (vhost_fds) {
s->vhostfd = vhost_fds[i];
@@ -1079,7 +1182,7 @@ int net_init_tap(const Netdev *netdev, const char *name,
if (!net_init_tap_one(tap, peer, name, ifname,
NULL, NULL,
vhost_fds ? vhost_fds[i] : -1,
- vnet_hdr, fds[i], errp)) {
+ vnet_hdr, fds[i], i, errp)) {
goto fail;
}
}
@@ -1113,7 +1216,7 @@ int net_init_tap(const Netdev *netdev, const char *name,
i >= 1 ? NULL : script,
i >= 1 ? NULL : downscript,
vhost_fds ? vhost_fds[i] : -1,
- vnet_hdr, fd, errp)) {
+ vnet_hdr, fd, i, errp)) {
goto fail;
}
}
@@ -1130,7 +1233,7 @@ fail:
int tap_enable(NetClientState *nc)
{
- TAPState *s = DO_UPCAST(TAPState, nc, nc);
+ TAPState *s = container_of(nc, TAPState, nc);
int ret;
if (s->enabled) {
@@ -1147,7 +1250,7 @@ int tap_enable(NetClientState *nc)
int tap_disable(NetClientState *nc)
{
- TAPState *s = DO_UPCAST(TAPState, nc, nc);
+ TAPState *s = container_of(nc, TAPState, nc);
int ret;
if (s->enabled == 0) {
@@ -1162,3 +1265,10 @@ int tap_disable(NetClientState *nc)
return ret;
}
}
+
+static void tap_register_types(void)
+{
+ type_register_static(&tap_netdev_info);
+}
+
+type_init(tap_register_types)
diff --git a/qapi/net.json b/qapi/net.json
index 82ddbb51cd7..029f5a4532c 100644
--- a/qapi/net.json
+++ b/qapi/net.json
@@ -429,9 +429,18 @@
# getting TAP file descriptors from incoming migration stream.
# The option is incompatible with any of @fd, @fds, @helper, @br,
# @ifname, @sndbuf and @vnet_hdr options, and requires @script and
-# @downscript be explicitly set to nothing (empty string or "no")
+# @downscript be explicitly set to nothing (empty string or "no"),
+# and requires also @local-migration to be set an "local"
+# migration parameter be set as well.
# (Since 11.1)
#
+# @local-migration: enable local migration for this TAP backend.
+# When set, local migration is enabled/disabled by "local"
+# migration parameter for this TAP backend. When unset, "local"
+# migration parameter is ignored for this TAP backend.
+# (Since 11.1. Defaults to true for MT >= 11.1,
+# and to false for MT < 11.1)
+#
# Since: 1.2
##
{ 'struct': 'NetdevTapOptions',
@@ -451,7 +460,8 @@
'*vhostforce': 'bool',
'*queues': 'uint32',
'*poll-us': 'uint32',
- '*incoming-fds': 'bool' } }
+ '*incoming-fds': 'bool',
+ '*local-migration': 'bool' } }
##
# @NetdevSocketOptions:
diff --git a/tests/functional/x86_64/test_tap_migration.py b/tests/functional/x86_64/test_tap_migration.py
index 7708facc153..b220969f9ad 100755
--- a/tests/functional/x86_64/test_tap_migration.py
+++ b/tests/functional/x86_64/test_tap_migration.py
@@ -332,6 +332,7 @@ def add_virtio_net(
"incoming-fds": incoming_fds,
"script": "no",
"downscript": "no",
+ "local-migration": local,
}
if not incoming_fds:
@@ -351,7 +352,6 @@ def add_virtio_net(
bus="pci.1",
mac=GUEST_MAC,
disable_legacy="off",
- local_migration=local,
)
def set_migration_capabilities(self, vm, local=True):
--
Best regards,
Vladimir
^ permalink raw reply related [flat|nested] 25+ messages in thread
* Re: [PATCH v16 5/8] virtio-net: support local migration of backend
2026-05-26 11:23 ` Vladimir Sementsov-Ogievskiy
@ 2026-05-26 17:47 ` Peter Xu
2026-05-27 7:24 ` Vladimir Sementsov-Ogievskiy
2026-05-27 11:35 ` Vladimir Sementsov-Ogievskiy
1 sibling, 1 reply; 25+ messages in thread
From: Peter Xu @ 2026-05-26 17:47 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy
Cc: Michael S. Tsirkin, jasowang, armbru, farosas, raphael.s.norwitz,
bchaney, qemu-devel, berrange, pbonzini, yc-core,
Philippe Mathieu-Daudé, Zhao Liu, Richard Henderson
On Tue, May 26, 2026 at 02:23:55PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> On 25.05.26 17:45, Peter Xu wrote:
> > On Mon, May 25, 2026 at 04:51:55PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> > > On 24.05.26 12:09, Michael S. Tsirkin wrote:
> > > > On Fri, May 22, 2026 at 03:05:30PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> > > > > Add virtio-net option local-migration, which is true by default,
> > > > > but false for older machine types, which doesn't support the feature.
> > > > >
> > > > > When both global migration parameter "local" and new virtio-net
> > > > > parameter "local-migration" are true, virtio-net transfer the whole
> > > > > net backend to the destination, including open file descriptors.
> > > > > Of-course, its only for local migration and the channel must be
> > > > > UNIX domain socket.
> > > > >
> > > > > This way management tool should not care about creating new TAP, and
> > > > > should not handle switching to it. Migration downtime become shorter.
> > > > >
> > > > > Support for TAP will come in the next commit.
> > > > >
> > > > > Signed-off-by: Vladimir Sementsov-Ogievskiy<vsementsov@yandex-team.ru>
> > > > > Reviewed-by: Ben Chaney<bchaney@akamai.com>
> > > > I don't get why is this a device property?
> > > > It's clearly a backend thing?
> > > >
> > > Hmm.
> > >
> > > We want to be able to disable fd-migration per device, when common "local"
> > > migration parameter is on.
> > >
> > > That's why we have parameter for virtio-net. It's also good, that we may
> > > enable it by default for newer machine types, to make fd-migration a default
> > > path.
> > >
> > > To be honest, it seems that the only thing in this patch (except for interface),
> > > which is not about backand, is that we want to do virtio_net_update_host_features()
> > > after backends incoming migration finished. For this, it's comfortable to have
> > > backend migration as part of device migration stream.
> > >
> > > --
> > >
> > > Imagine, we move "local-migration" parameter to TAP device. This raise a lot of questions:
> > Hmm, I thought we discussed this quite some time ago.. I can't remember
> > details, but I'll try to comment with what I can still remember.
> >
> > > 1. We loss a possibility to set good default in machine type. Ok, we probably may add
> > > a property to "migration" object, but this property will look like "TAP-local-migration",
> > > and raise discussion again, that it should be property of TAP..
> > If you recall I worked on the other series because of this desire of having
> > tap be able to be QOMified and also support machine compat properties:
> >
> > https://lore.kernel.org/all/20251209162857.857593-2-peterx@redhat.com/
> >
> > I didn't get a lot of feedback supporting having TYPE_OBJECT_COMPAT, maybe
> > it's an overkill only to apply object_apply_compat_props().
> >
> > However, just to say we can still QOMify TAP and then add its own
> > instance_post_init() to also do object_apply_compat_props(), like quite a
> > few other existing users, then this (1) isn't a problem, and it doesn't
> > need to depend on my series either.
>
> Thanks for reminding and explanation. Previously, I thought that
> QOMification is huge task, and your series is required. Now I see,
> that it is not so difficult. All that's left for me to say is
> "let's try it")
>
>
> Here is an experimental diff on top on this series. If you like it,
> I'll resend with proper division into commits:
I don't know networking, but in general it looks pretty neat to me, thanks.
Only a few nitpicks inline where I can spot.
>
>
> diff --git a/hw/core/machine.c b/hw/core/machine.c
> index 619e80c1cb3..67e873c74fe 100644
> --- a/hw/core/machine.c
> +++ b/hw/core/machine.c
> @@ -36,12 +36,13 @@
> #include "hw/virtio/virtio-pci.h"
> #include "hw/virtio/virtio-net.h"
> #include "hw/virtio/virtio-iommu.h"
> +#include "net/tap.h"
> #include "hw/acpi/generic_event_device.h"
> #include "qemu/audio.h"
> GlobalProperty hw_compat_11_0[] = {
> { "chardev-vc", "encoding", "cp437" },
> - { TYPE_VIRTIO_NET, "local-migration", "false" },
> + { TYPE_TAP_NETDEV, "local-migration", "false" },
> };
> const size_t hw_compat_11_0_len = G_N_ELEMENTS(hw_compat_11_0);
> diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
> index 158b9247a58..bad349d3e83 100644
> --- a/hw/net/virtio-net.c
> +++ b/hw/net/virtio-net.c
> @@ -3202,24 +3202,13 @@ static void virtio_net_get_features(VirtIODevice *vdev, uint64_t *features,
> }
> }
> -static bool virtio_net_update_host_features(VirtIONet *n, Error **errp)
> -{
> - ERRP_GUARD();
> - VirtIODevice *vdev = VIRTIO_DEVICE(n);
> -
> - peer_test_vnet_hdr(n);
> -
> - virtio_net_get_features(vdev, &vdev->host_features, errp);
> -
> - return !*errp;
> -}
> -
> static int virtio_net_post_load_device(void *opaque, int version_id)
> {
> VirtIONet *n = opaque;
> VirtIODevice *vdev = VIRTIO_DEVICE(n);
> int i, link_down;
> bool has_tunnel_hdr = virtio_has_tunnel_hdr(vdev->guest_features_ex);
> + Error *local_err = NULL;
> trace_virtio_net_post_load_device();
> virtio_net_set_mrg_rx_bufs(n, n->mergeable_rx_bufs,
> @@ -3277,6 +3266,19 @@ static int virtio_net_post_load_device(void *opaque, int version_id)
> }
> virtio_net_commit_rss_config(n);
> +
> + /*
> + * The TAP backend has already been migrated at higher priority
> + * (MIG_PRI_TAP) and virtio_net_vnet_post_load() has already called
> + * peer_test_vnet_hdr(). Recompute host_features so that virtio-net
> + * reflects the capabilities of the restored backend.
> + */
> + virtio_net_get_features(vdev, &vdev->host_features, &local_err);
> + if (local_err) {
> + error_report_err(local_err);
> + return -EINVAL;
> + }
> +
> return 0;
> }
> @@ -3430,6 +3432,13 @@ static int virtio_net_vnet_post_load(void *opaque, int version_id)
> {
> struct VirtIONetMigTmp *tmp = opaque;
> + /*
> + * The TAP backend has already been migrated at higher priority
> + * (MIG_PRI_TAP), so n->has_vnet_hdr can be refreshed from the live
> + * backend right here.
> + */
> + peer_test_vnet_hdr(tmp->parent);
> +
> if (tmp->has_vnet_hdr && !peer_has_vnet_hdr(tmp->parent)) {
> error_report("virtio-net: saved image requires vnet_hdr=on");
> return -EINVAL;
> @@ -3607,57 +3616,6 @@ static const VMStateDescription vhost_user_net_backend_state = {
> }
> };
> -static bool virtio_net_migrate_local(void *opaque, int version_id)
> -{
> - VirtIONet *n = opaque;
> -
> - return migrate_local() && n->local_migration;
> -}
> -
> -static int virtio_net_nic_pre_save(void *opaque)
> -{
> - struct VirtIONetMigTmp *tmp = opaque;
> -
> - tmp->ncs = tmp->parent->nic->ncs;
> - tmp->max_queue_pairs = tmp->parent->max_queue_pairs;
> -
> - return 0;
> -}
> -
> -static int virtio_net_nic_pre_load(void *opaque)
> -{
> - /* Reuse the pointer setup from save */
> - virtio_net_nic_pre_save(opaque);
> -
> - return 0;
> -}
> -
> -static int virtio_net_nic_post_load(void *opaque, int version_id)
> -{
> - struct VirtIONetMigTmp *tmp = opaque;
> - Error *local_err = NULL;
> -
> - if (!virtio_net_update_host_features(tmp->parent, &local_err)) {
> - error_report_err(local_err);
> - return -EINVAL;
> - }
> -
> - return 0;
> -}
> -
> -static const VMStateDescription vmstate_virtio_net_nic = {
> - .name = "virtio-net-nic",
> - .pre_load = virtio_net_nic_pre_load,
> - .pre_save = virtio_net_nic_pre_save,
> - .post_load = virtio_net_nic_post_load,
> - .fields = (const VMStateField[]) {
> - VMSTATE_VARRAY_UINT32(ncs, struct VirtIONetMigTmp,
> - max_queue_pairs, 0, vmstate_net_peer_backend,
> - NetClientState),
> - VMSTATE_END_OF_LIST()
> - },
> -};
> -
> static const VMStateDescription vmstate_virtio_net_device = {
> .name = "virtio-net-device",
> .version_id = VIRTIO_NET_VM_VERSION,
> @@ -3689,9 +3647,6 @@ static const VMStateDescription vmstate_virtio_net_device = {
> * but based on the uint.
> */
> VMSTATE_BUFFER_POINTER_UNSAFE(vlans, VirtIONet, 0, MAX_VLAN >> 3),
> - VMSTATE_WITH_TMP_TEST(VirtIONet, virtio_net_migrate_local,
> - struct VirtIONetMigTmp,
> - vmstate_virtio_net_nic),
> VMSTATE_WITH_TMP(VirtIONet, struct VirtIONetMigTmp,
> vmstate_virtio_net_has_vnet),
> VMSTATE_UINT8(mac_table.multi_overflow, VirtIONet),
> @@ -4444,7 +4399,6 @@ static const Property virtio_net_properties[] = {
> host_features_ex,
> VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO_CSUM,
> true),
> - DEFINE_PROP_BOOL("local-migration", VirtIONet, local_migration, true),
> };
> static void virtio_net_class_init(ObjectClass *klass, const void *data)
> diff --git a/include/hw/virtio/virtio-net.h b/include/hw/virtio/virtio-net.h
> index 0c14e314409..8c967760c2a 100644
> --- a/include/hw/virtio/virtio-net.h
> +++ b/include/hw/virtio/virtio-net.h
> @@ -231,7 +231,6 @@ struct VirtIONet {
> uint32_t nr_ebpf_rss_fds;
> char **ebpf_rss_fds;
> bool peers_wait_incoming;
> - bool local_migration;
> };
> size_t virtio_net_handle_ctrl_iov(VirtIODevice *vdev,
> diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h
> index 0a8a2e85a63..e7b3cd7db01 100644
> --- a/include/migration/vmstate.h
> +++ b/include/migration/vmstate.h
> @@ -178,6 +178,7 @@ typedef enum {
> MIG_PRI_LOW, /* Must happen after default */
> MIG_PRI_DEFAULT,
> + MIG_PRI_TAP, /* Must happen before virtio-net device */
It's a tradition we put random names here, but they should be cleaned up at
some point. For this one, it's easy to categorize it on the 1st day, maybe
MIG_PRI_BACKEND? Which should include anything that is a backend of an
emulated devices. Keeping the comment would be still very nice, though (by
mentioning the dependency of virtio-net over TAP).
> MIG_PRI_IOMMU, /* Must happen before PCI devices */
> MIG_PRI_PCI_BUS, /* Must happen before IOMMU */
> MIG_PRI_VIRTIO_MEM, /* Must happen before IOMMU */
> diff --git a/include/net/net.h b/include/net/net.h
> index d4cf399d4a8..be7ca0ce845 100644
> --- a/include/net/net.h
> +++ b/include/net/net.h
> @@ -113,7 +113,6 @@ typedef struct NetClientInfo {
> NetCheckPeerType *check_peer_type;
> IsWaitIncoming *is_wait_incoming;
> GetVHostNet *get_vhost_net;
> - const VMStateDescription *backend_vmsd;
> } NetClientInfo;
> struct NetClientState {
> @@ -164,6 +163,13 @@ char *qemu_mac_strdup_printf(const uint8_t *macaddr);
> NetClientState *qemu_find_netdev(const char *id);
> int qemu_find_net_clients_except(const char *id, NetClientState **ncs,
> NetClientDriver type, int max);
> +void qemu_net_client_setup(NetClientState *nc,
> + NetClientInfo *info,
> + NetClientState *peer,
> + const char *model,
> + const char *name,
> + NetClientDestructor *destructor,
> + bool is_datapath);
> NetClientState *qemu_new_net_client(NetClientInfo *info,
> NetClientState *peer,
> const char *model,
> @@ -358,6 +364,4 @@ static inline bool net_peer_needs_padding(NetClientState *nc)
> return nc->peer && !nc->peer->do_not_pad;
> }
> -extern const VMStateInfo vmstate_net_peer_backend;
> -
> #endif
> diff --git a/include/net/tap.h b/include/net/tap.h
> index 6f34f13eae4..325de81f03b 100644
> --- a/include/net/tap.h
> +++ b/include/net/tap.h
> @@ -28,9 +28,12 @@
> #include "standard-headers/linux/virtio_net.h"
> +#define TYPE_TAP_NETDEV "tap-netdev"
> +
> int tap_enable(NetClientState *nc);
> int tap_disable(NetClientState *nc);
> int tap_get_fd(NetClientState *nc);
> +bool tap_get_local_migration(NetClientState *nc);
> #endif /* QEMU_NET_TAP_H */
> diff --git a/net/net.c b/net/net.c
> index 8bccb1880a1..34766763ae5 100644
> --- a/net/net.c
> +++ b/net/net.c
> @@ -58,7 +58,6 @@
> #include "qapi/string-output-visitor.h"
> #include "qapi/qobject-input-visitor.h"
> #include "standard-headers/linux/virtio_net.h"
> -#include "migration/vmstate.h"
> /* Net bridge is currently not supported for W32. */
> #if !defined(_WIN32)
> @@ -262,13 +261,13 @@ static ssize_t qemu_deliver_packet_iov(NetClientState *sender,
> int iovcnt,
> void *opaque);
> -static void qemu_net_client_setup(NetClientState *nc,
> - NetClientInfo *info,
> - NetClientState *peer,
> - const char *model,
> - const char *name,
> - NetClientDestructor *destructor,
> - bool is_datapath)
> +void qemu_net_client_setup(NetClientState *nc,
> + NetClientInfo *info,
> + NetClientState *peer,
> + const char *model,
> + const char *name,
> + NetClientDestructor *destructor,
> + bool is_datapath)
> {
> nc->info = info;
> nc->model = g_strdup(model);
> @@ -2175,48 +2174,3 @@ int net_fill_rstate(SocketReadState *rs, const uint8_t *buf, int size)
> return 0;
> }
> -static int get_peer_backend(QEMUFile *f, void *pv, size_t size,
> - const VMStateField *field)
> -{
> - NetClientState *nc = pv;
> - Error *local_err = NULL;
> - int ret;
> -
> - if (!nc->peer) {
> - return -EINVAL;
> - }
> - nc = nc->peer;
> -
> - ret = vmstate_load_state(f, nc->info->backend_vmsd, nc, 0, &local_err);
> - if (ret < 0) {
> - error_report_err(local_err);
> - }
> -
> - return ret;
> -}
> -
> -static int put_peer_backend(QEMUFile *f, void *pv, size_t size,
> - const VMStateField *field, JSONWriter *vmdesc)
> -{
> - NetClientState *nc = pv;
> - Error *local_err = NULL;
> - int ret;
> -
> - if (!nc->peer) {
> - return -EINVAL;
> - }
> - nc = nc->peer;
> -
> - ret = vmstate_save_state(f, nc->info->backend_vmsd, nc, 0, &local_err);
> - if (ret < 0) {
> - error_report_err(local_err);
> - }
> -
> - return ret;
> -}
> -
> -const VMStateInfo vmstate_net_peer_backend = {
> - .name = "virtio-net-nic-nc-backend",
> - .get = get_peer_backend,
> - .put = put_peer_backend,
> -};
> diff --git a/net/tap.c b/net/tap.c
> index 9b1d4613a0b..e3aeb99d249 100644
> --- a/net/tap.c
> +++ b/net/tap.c
> @@ -38,12 +38,17 @@
> #include "monitor/monitor.h"
> #include "system/runstate.h"
> #include "system/system.h"
> +#include "migration/misc.h"
> #include "qapi/error.h"
> #include "qemu/cutils.h"
> #include "qemu/error-report.h"
> #include "qemu/main-loop.h"
> #include "qemu/sockets.h"
> #include "hw/virtio/vhost.h"
> +#include "hw/core/vmstate-if.h"
> +#include "migration/vmstate.h"
> +#include "qom/object.h"
> +#include "qom/compat-properties.h"
> #include "net/tap.h"
> #include "net/util.h"
> @@ -69,7 +74,13 @@ static const int kernel_feature_bits[] = {
> VHOST_INVALID_FEATURE_BIT
> };
> -typedef struct TAPState {
> +OBJECT_DECLARE_SIMPLE_TYPE(TAPState, TAP_NETDEV)
> +
> +static const VMStateDescription vmstate_tap;
> +
> +struct TAPState {
> + Object parent_obj;
> +
> NetClientState nc;
> int fd;
> int vhostfd;
> @@ -90,7 +101,9 @@ typedef struct TAPState {
> bool read_poll_detached;
> VMChangeStateEntry *vmstate;
> -} TAPState;
> + bool local_migration;
> + int queue_index;
> +};
> static void launch_script(const char *setup_script, const char *ifname,
> int fd, Error **errp);
> @@ -182,7 +195,7 @@ static ssize_t tap_write_packet(TAPState *s, const struct iovec *iov, int iovcnt
> static ssize_t tap_receive_iov(NetClientState *nc, const struct iovec *iov,
> int iovcnt)
> {
> - TAPState *s = DO_UPCAST(TAPState, nc, nc);
> + TAPState *s = container_of(nc, TAPState, nc);
> const struct iovec *iovp = iov;
> g_autofree struct iovec *iov_copy = NULL;
> struct virtio_net_hdr hdr = { };
> @@ -218,7 +231,7 @@ ssize_t tap_read_packet(int tapfd, uint8_t *buf, int maxlen)
> static void tap_send_completed(NetClientState *nc, ssize_t len)
> {
> - TAPState *s = DO_UPCAST(TAPState, nc, nc);
> + TAPState *s = container_of(nc, TAPState, nc);
> tap_read_poll(s, true);
> }
> @@ -278,7 +291,7 @@ static void tap_send(void *opaque)
> static bool tap_has_ufo(NetClientState *nc)
> {
> - TAPState *s = DO_UPCAST(TAPState, nc, nc);
> + TAPState *s = container_of(nc, TAPState, nc);
> assert(nc->info->type == NET_CLIENT_DRIVER_TAP);
> @@ -287,7 +300,7 @@ static bool tap_has_ufo(NetClientState *nc)
> static bool tap_has_uso(NetClientState *nc)
> {
> - TAPState *s = DO_UPCAST(TAPState, nc, nc);
> + TAPState *s = container_of(nc, TAPState, nc);
> assert(nc->info->type == NET_CLIENT_DRIVER_TAP);
> @@ -296,7 +309,7 @@ static bool tap_has_uso(NetClientState *nc)
> static bool tap_has_tunnel(NetClientState *nc)
> {
> - TAPState *s = DO_UPCAST(TAPState, nc, nc);
> + TAPState *s = container_of(nc, TAPState, nc);
> assert(nc->info->type == NET_CLIENT_DRIVER_TAP);
> return s->has_tunnel;
> @@ -304,7 +317,7 @@ static bool tap_has_tunnel(NetClientState *nc)
> static bool tap_has_vnet_hdr(NetClientState *nc)
> {
> - TAPState *s = DO_UPCAST(TAPState, nc, nc);
> + TAPState *s = container_of(nc, TAPState, nc);
> assert(nc->info->type == NET_CLIENT_DRIVER_TAP);
> @@ -318,7 +331,7 @@ static bool tap_has_vnet_hdr_len(NetClientState *nc, int len)
> static void tap_set_vnet_hdr_len(NetClientState *nc, int len)
> {
> - TAPState *s = DO_UPCAST(TAPState, nc, nc);
> + TAPState *s = container_of(nc, TAPState, nc);
> assert(nc->info->type == NET_CLIENT_DRIVER_TAP);
> @@ -329,21 +342,21 @@ static void tap_set_vnet_hdr_len(NetClientState *nc, int len)
> static int tap_set_vnet_le(NetClientState *nc, bool is_le)
> {
> - TAPState *s = DO_UPCAST(TAPState, nc, nc);
> + TAPState *s = container_of(nc, TAPState, nc);
> return tap_fd_set_vnet_le(s->fd, is_le);
> }
> static int tap_set_vnet_be(NetClientState *nc, bool is_be)
> {
> - TAPState *s = DO_UPCAST(TAPState, nc, nc);
> + TAPState *s = container_of(nc, TAPState, nc);
> return tap_fd_set_vnet_be(s->fd, is_be);
> }
> static void tap_set_offload(NetClientState *nc, const NetOffloads *ol)
> {
> - TAPState *s = DO_UPCAST(TAPState, nc, nc);
> + TAPState *s = container_of(nc, TAPState, nc);
> if (s->fd < 0) {
> return;
> }
> @@ -364,7 +377,7 @@ static void tap_exit_notify(Notifier *notifier, void *data)
> static void tap_cleanup(NetClientState *nc)
> {
> - TAPState *s = DO_UPCAST(TAPState, nc, nc);
> + TAPState *s = container_of(nc, TAPState, nc);
> if (s->vhost_net) {
> vhost_net_cleanup(s->vhost_net);
> @@ -389,18 +402,20 @@ static void tap_cleanup(NetClientState *nc)
> tap_write_poll(s, false);
> close(s->fd);
> s->fd = -1;
> +
> + vmstate_unregister(VMSTATE_IF(s), &vmstate_tap, s);
> }
> static void tap_poll(NetClientState *nc, bool enable)
> {
> - TAPState *s = DO_UPCAST(TAPState, nc, nc);
> + TAPState *s = container_of(nc, TAPState, nc);
> tap_read_poll(s, enable);
> tap_write_poll(s, enable);
> }
> static bool tap_set_steering_ebpf(NetClientState *nc, int prog_fd)
> {
> - TAPState *s = DO_UPCAST(TAPState, nc, nc);
> + TAPState *s = container_of(nc, TAPState, nc);
> assert(nc->info->type == NET_CLIENT_DRIVER_TAP);
> return tap_fd_set_steering_ebpf(s->fd, prog_fd) == 0;
> @@ -408,7 +423,7 @@ static bool tap_set_steering_ebpf(NetClientState *nc, int prog_fd)
> int tap_get_fd(NetClientState *nc)
> {
> - TAPState *s = DO_UPCAST(TAPState, nc, nc);
> + TAPState *s = container_of(nc, TAPState, nc);
> assert(nc->info->type == NET_CLIENT_DRIVER_TAP);
> return s->fd;
> }
> @@ -420,14 +435,14 @@ int tap_get_fd(NetClientState *nc)
> */
> static VHostNetState *tap_get_vhost_net(NetClientState *nc)
> {
> - TAPState *s = DO_UPCAST(TAPState, nc, nc);
> + TAPState *s = container_of(nc, TAPState, nc);
> assert(nc->info->type == NET_CLIENT_DRIVER_TAP);
> return s->vhost_net;
> }
> static bool tap_is_wait_incoming(NetClientState *nc)
> {
> - TAPState *s = DO_UPCAST(TAPState, nc, nc);
> + TAPState *s = container_of(nc, TAPState, nc);
> assert(nc->info->type == NET_CLIENT_DRIVER_TAP);
> return s->fd == -1;
> }
> @@ -468,10 +483,19 @@ static int tap_post_load(void *opaque, int version_id)
> return 0;
> }
> +static bool tap_needed(void *opaque)
> +{
> + TAPState *s = opaque;
> +
> + return s->local_migration && migrate_local();
> +}
> +
> static const VMStateDescription vmstate_tap = {
> .name = "net-tap",
> + .priority = MIG_PRI_TAP,
> .pre_load = tap_pre_load,
> .post_load = tap_post_load,
> + .needed = tap_needed,
> .fields = (const VMStateField[]) {
> VMSTATE_FD(fd, TAPState),
> VMSTATE_BOOL(using_vnet_hdr, TAPState),
> @@ -484,6 +508,63 @@ static const VMStateDescription vmstate_tap = {
> }
> };
> +static char *tap_vmstate_if_get_id(VMStateIf *obj)
> +{
> + TAPState *s = TAP_NETDEV(obj);
> + char *res = g_strdup_printf("%s/%d", s->nc.name, s->queue_index);
> + return res;
> +}
> +
> +static bool tap_get_local_migration_prop(Object *obj, Error **errp)
> +{
> + TAPState *s = TAP_NETDEV(obj);
> + return s->local_migration;
> +}
> +
> +static void tap_set_local_migration_prop(Object *obj, bool value, Error **errp)
> +{
> + TAPState *s = TAP_NETDEV(obj);
> + s->local_migration = value;
> +}
> +
> +static void tap_instance_init(Object *obj)
> +{
> + TAPState *s = TAP_NETDEV(obj);
> + s->local_migration = true;
> +}
> +
> +static void tap_class_init(ObjectClass *klass, const void *data)
> +{
> + VMStateIfClass *vc = VMSTATE_IF_CLASS(klass);
> +
> + vc->get_id = tap_vmstate_if_get_id;
> +
> + object_class_property_add_bool(klass, "local-migration",
> + tap_get_local_migration_prop,
> + tap_set_local_migration_prop);
> + object_class_property_set_description(klass, "local-migration",
> + "Migrate the TAP file descriptor to the target (local migration)");
> +}
> +
> +static const TypeInfo tap_netdev_info = {
> + .name = TYPE_TAP_NETDEV,
> + .parent = TYPE_OBJECT,
> + .instance_size = sizeof(TAPState),
> + .instance_init = tap_instance_init,
> + .instance_post_init = object_apply_compat_props,
> + .class_init = tap_class_init,
> + .interfaces = (const InterfaceInfo[]) {
> + { TYPE_VMSTATE_IF },
> + { }
> + },
> +};
> +
> +static void tap_net_client_destructor(NetClientState *nc)
> +{
> + TAPState *s = container_of(nc, TAPState, nc);
> + object_unref(OBJECT(s));
> +}
> +
> /* fd support */
> static NetClientInfo net_tap_info = {
> @@ -505,22 +586,43 @@ static NetClientInfo net_tap_info = {
> .set_steering_ebpf = tap_set_steering_ebpf,
> .is_wait_incoming = tap_is_wait_incoming,
> .get_vhost_net = tap_get_vhost_net,
> - .backend_vmsd = &vmstate_tap,
> };
> +static TAPState *new_tap(NetClientState *peer,
> + const char *model,
> + const char *name,
> + int queue_index,
> + bool has_local_migration,
> + bool local_migration)
> +{
> + TAPState *s = TAP_NETDEV(object_new(TYPE_TAP_NETDEV));
> +
> + qemu_net_client_setup(&s->nc, &net_tap_info, peer, model, name,
> + tap_net_client_destructor, true);
> +
> + s->queue_index = queue_index;
> +
> + if (has_local_migration) {
> + s->local_migration = local_migration;
> + }
> +
> + vmstate_register(VMSTATE_IF(s), VMSTATE_INSTANCE_ID_ANY, &vmstate_tap, s);
> +
> + return s;
> +}
> +
> static TAPState *net_tap_fd_init(NetClientState *peer,
> const char *model,
> const char *name,
> int fd,
> - int vnet_hdr)
> + int vnet_hdr,
> + int queue_index,
> + bool has_local_migration,
> + bool local_migration)
> {
> NetOffloads ol = {};
> - NetClientState *nc;
> - TAPState *s;
> -
> - nc = qemu_new_net_client(&net_tap_info, peer, model, name);
> -
> - s = DO_UPCAST(TAPState, nc, nc);
> + TAPState *s = new_tap(peer, model, name, queue_index,
> + has_local_migration, local_migration);
> s->fd = fd;
> s->host_vnet_hdr_len = vnet_hdr ? sizeof(struct virtio_net_hdr) : 0;
> @@ -757,7 +859,7 @@ int net_init_bridge(const Netdev *netdev, const char *name,
> close(fd);
> return -1;
> }
> - s = net_tap_fd_init(peer, "bridge", name, fd, vnet_hdr);
> + s = net_tap_fd_init(peer, "bridge", name, fd, vnet_hdr, 0, true, false);
> qemu_set_info_str(&s->nc, "helper=%s,br=%s", helper, br);
> @@ -833,10 +935,13 @@ static bool net_init_tap_one(const NetdevTapOptions *tap, NetClientState *peer,
> const char *name,
> const char *ifname, const char *script,
> const char *downscript, int vhostfd,
> - int vnet_hdr, int fd, Error **errp)
> + int vnet_hdr, int fd, int queue_index,
> + Error **errp)
> {
> TAPState *s = net_tap_fd_init(peer, tap->helper ? "bridge" : "tap",
> - name, fd, vnet_hdr);
> + name, fd, vnet_hdr, queue_index,
> + tap->has_local_migration,
> + tap->local_migration);
> bool sndbuf_required = tap->has_sndbuf;
> int sndbuf =
> (tap->has_sndbuf && tap->sndbuf) ? MIN(tap->sndbuf, INT_MAX) : INT_MAX;
> @@ -1048,13 +1153,11 @@ int net_init_tap(const Netdev *netdev, const char *name,
> if (tap->incoming_fds) {
> for (i = 0; i < queues; i++) {
> - NetClientState *nc;
> - TAPState *s;
> + TAPState *s = new_tap(peer, "tap", name, i,
> + tap->has_local_migration, tap->local_migration);
> - nc = qemu_new_net_client(&net_tap_info, peer, "tap", name);
> - qemu_set_info_str(nc, "incoming");
> + qemu_set_info_str(&s->nc, "incoming");
> - s = DO_UPCAST(TAPState, nc, nc);
> s->fd = -1;
> if (vhost_fds) {
> s->vhostfd = vhost_fds[i];
> @@ -1079,7 +1182,7 @@ int net_init_tap(const Netdev *netdev, const char *name,
> if (!net_init_tap_one(tap, peer, name, ifname,
> NULL, NULL,
> vhost_fds ? vhost_fds[i] : -1,
> - vnet_hdr, fds[i], errp)) {
> + vnet_hdr, fds[i], i, errp)) {
> goto fail;
> }
> }
> @@ -1113,7 +1216,7 @@ int net_init_tap(const Netdev *netdev, const char *name,
> i >= 1 ? NULL : script,
> i >= 1 ? NULL : downscript,
> vhost_fds ? vhost_fds[i] : -1,
> - vnet_hdr, fd, errp)) {
> + vnet_hdr, fd, i, errp)) {
> goto fail;
> }
> }
> @@ -1130,7 +1233,7 @@ fail:
> int tap_enable(NetClientState *nc)
> {
> - TAPState *s = DO_UPCAST(TAPState, nc, nc);
> + TAPState *s = container_of(nc, TAPState, nc);
> int ret;
> if (s->enabled) {
> @@ -1147,7 +1250,7 @@ int tap_enable(NetClientState *nc)
> int tap_disable(NetClientState *nc)
> {
> - TAPState *s = DO_UPCAST(TAPState, nc, nc);
> + TAPState *s = container_of(nc, TAPState, nc);
> int ret;
> if (s->enabled == 0) {
> @@ -1162,3 +1265,10 @@ int tap_disable(NetClientState *nc)
> return ret;
> }
> }
> +
> +static void tap_register_types(void)
> +{
> + type_register_static(&tap_netdev_info);
> +}
> +
> +type_init(tap_register_types)
> diff --git a/qapi/net.json b/qapi/net.json
> index 82ddbb51cd7..029f5a4532c 100644
> --- a/qapi/net.json
> +++ b/qapi/net.json
> @@ -429,9 +429,18 @@
> # getting TAP file descriptors from incoming migration stream.
> # The option is incompatible with any of @fd, @fds, @helper, @br,
> # @ifname, @sndbuf and @vnet_hdr options, and requires @script and
> -# @downscript be explicitly set to nothing (empty string or "no")
> +# @downscript be explicitly set to nothing (empty string or "no"),
> +# and requires also @local-migration to be set an "local"
> +# migration parameter be set as well.
> # (Since 11.1)
> #
> +# @local-migration: enable local migration for this TAP backend.
I would maybe call it "supported", then we can differenciate "enablement of
the feature" versus "devices support this feature".
Not a big deal, though. Below lines explain everything well, so looks good
with/without changing.
> +# When set, local migration is enabled/disabled by "local"
> +# migration parameter for this TAP backend. When unset, "local"
> +# migration parameter is ignored for this TAP backend.
> +# (Since 11.1. Defaults to true for MT >= 11.1,
> +# and to false for MT < 11.1)
> +#
> # Since: 1.2
> ##
> { 'struct': 'NetdevTapOptions',
> @@ -451,7 +460,8 @@
> '*vhostforce': 'bool',
> '*queues': 'uint32',
> '*poll-us': 'uint32',
> - '*incoming-fds': 'bool' } }
> + '*incoming-fds': 'bool',
> + '*local-migration': 'bool' } }
> ##
> # @NetdevSocketOptions:
> diff --git a/tests/functional/x86_64/test_tap_migration.py b/tests/functional/x86_64/test_tap_migration.py
> index 7708facc153..b220969f9ad 100755
> --- a/tests/functional/x86_64/test_tap_migration.py
> +++ b/tests/functional/x86_64/test_tap_migration.py
> @@ -332,6 +332,7 @@ def add_virtio_net(
> "incoming-fds": incoming_fds,
> "script": "no",
> "downscript": "no",
> + "local-migration": local,
> }
> if not incoming_fds:
> @@ -351,7 +352,6 @@ def add_virtio_net(
> bus="pci.1",
> mac=GUEST_MAC,
> disable_legacy="off",
> - local_migration=local,
> )
> def set_migration_capabilities(self, vm, local=True):
>
>
> --
> Best regards,
> Vladimir
>
--
Peter Xu
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v16 5/8] virtio-net: support local migration of backend
2026-05-26 17:47 ` Peter Xu
@ 2026-05-27 7:24 ` Vladimir Sementsov-Ogievskiy
0 siblings, 0 replies; 25+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2026-05-27 7:24 UTC (permalink / raw)
To: Peter Xu
Cc: Michael S. Tsirkin, jasowang, armbru, farosas, raphael.s.norwitz,
bchaney, qemu-devel, berrange, pbonzini, yc-core,
Philippe Mathieu-Daudé, Zhao Liu, Richard Henderson
On 26.05.26 20:47, Peter Xu wrote:
> On Tue, May 26, 2026 at 02:23:55PM +0300, Vladimir Sementsov-Ogievskiy wrote:
>> On 25.05.26 17:45, Peter Xu wrote:
>>> On Mon, May 25, 2026 at 04:51:55PM +0300, Vladimir Sementsov-Ogievskiy wrote:
>>>> On 24.05.26 12:09, Michael S. Tsirkin wrote:
>>>>> On Fri, May 22, 2026 at 03:05:30PM +0300, Vladimir Sementsov-Ogievskiy wrote:
>>>>>> Add virtio-net option local-migration, which is true by default,
>>>>>> but false for older machine types, which doesn't support the feature.
>>>>>>
>>>>>> When both global migration parameter "local" and new virtio-net
>>>>>> parameter "local-migration" are true, virtio-net transfer the whole
>>>>>> net backend to the destination, including open file descriptors.
>>>>>> Of-course, its only for local migration and the channel must be
>>>>>> UNIX domain socket.
>>>>>>
>>>>>> This way management tool should not care about creating new TAP, and
>>>>>> should not handle switching to it. Migration downtime become shorter.
>>>>>>
>>>>>> Support for TAP will come in the next commit.
>>>>>>
>>>>>> Signed-off-by: Vladimir Sementsov-Ogievskiy<vsementsov@yandex-team.ru>
>>>>>> Reviewed-by: Ben Chaney<bchaney@akamai.com>
>>>>> I don't get why is this a device property?
>>>>> It's clearly a backend thing?
>>>>>
>>>> Hmm.
>>>>
>>>> We want to be able to disable fd-migration per device, when common "local"
>>>> migration parameter is on.
>>>>
>>>> That's why we have parameter for virtio-net. It's also good, that we may
>>>> enable it by default for newer machine types, to make fd-migration a default
>>>> path.
>>>>
>>>> To be honest, it seems that the only thing in this patch (except for interface),
>>>> which is not about backand, is that we want to do virtio_net_update_host_features()
>>>> after backends incoming migration finished. For this, it's comfortable to have
>>>> backend migration as part of device migration stream.
>>>>
>>>> --
>>>>
>>>> Imagine, we move "local-migration" parameter to TAP device. This raise a lot of questions:
>>> Hmm, I thought we discussed this quite some time ago.. I can't remember
>>> details, but I'll try to comment with what I can still remember.
>>>
>>>> 1. We loss a possibility to set good default in machine type. Ok, we probably may add
>>>> a property to "migration" object, but this property will look like "TAP-local-migration",
>>>> and raise discussion again, that it should be property of TAP..
>>> If you recall I worked on the other series because of this desire of having
>>> tap be able to be QOMified and also support machine compat properties:
>>>
>>> https://lore.kernel.org/all/20251209162857.857593-2-peterx@redhat.com/
>>>
>>> I didn't get a lot of feedback supporting having TYPE_OBJECT_COMPAT, maybe
>>> it's an overkill only to apply object_apply_compat_props().
>>>
>>> However, just to say we can still QOMify TAP and then add its own
>>> instance_post_init() to also do object_apply_compat_props(), like quite a
>>> few other existing users, then this (1) isn't a problem, and it doesn't
>>> need to depend on my series either.
>>
>> Thanks for reminding and explanation. Previously, I thought that
>> QOMification is huge task, and your series is required. Now I see,
>> that it is not so difficult. All that's left for me to say is
>> "let's try it")
>>
>>
>> Here is an experimental diff on top on this series. If you like it,
>> I'll resend with proper division into commits:
>
> I don't know networking, but in general it looks pretty neat to me, thanks.
>
> Only a few nitpicks inline where I can spot.
[..]
>> diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h
>> index 0a8a2e85a63..e7b3cd7db01 100644
>> --- a/include/migration/vmstate.h
>> +++ b/include/migration/vmstate.h
>> @@ -178,6 +178,7 @@ typedef enum {
>> MIG_PRI_LOW, /* Must happen after default */
>> MIG_PRI_DEFAULT,
>> + MIG_PRI_TAP, /* Must happen before virtio-net device */
>
> It's a tradition we put random names here, but they should be cleaned up at
> some point. For this one, it's easy to categorize it on the 1st day, maybe
> MIG_PRI_BACKEND? Which should include anything that is a backend of an
> emulated devices. Keeping the comment would be still very nice, though (by
> mentioning the dependency of virtio-net over TAP).
OK
>
>> MIG_PRI_IOMMU, /* Must happen before PCI devices */
>> MIG_PRI_PCI_BUS, /* Must happen before IOMMU */
>> MIG_PRI_VIRTIO_MEM, /* Must happen before IOMMU */
[..]
>> --- a/qapi/net.json
>> +++ b/qapi/net.json
>> @@ -429,9 +429,18 @@
>> # getting TAP file descriptors from incoming migration stream.
>> # The option is incompatible with any of @fd, @fds, @helper, @br,
>> # @ifname, @sndbuf and @vnet_hdr options, and requires @script and
>> -# @downscript be explicitly set to nothing (empty string or "no")
>> +# @downscript be explicitly set to nothing (empty string or "no"),
>> +# and requires also @local-migration to be set an "local"
>> +# migration parameter be set as well.
>> # (Since 11.1)
>> #
>> +# @local-migration: enable local migration for this TAP backend.
>
> I would maybe call it "supported", then we can differenciate "enablement of
> the feature" versus "devices support this feature".
>
> Not a big deal, though. Below lines explain everything well, so looks good
> with/without changing.
OK for me, will do.
Thanks for fast answer, I'll make v17 soon.
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v16 0/8] virtio-net: live-TAP local migration
2026-05-22 12:05 [PATCH v16 0/8] virtio-net: live-TAP local migration Vladimir Sementsov-Ogievskiy
` (7 preceding siblings ...)
2026-05-22 12:05 ` [PATCH v16 8/8] tests/functional: add test_tap_migration Vladimir Sementsov-Ogievskiy
@ 2026-05-27 10:46 ` Mark Cave-Ayland
2026-05-27 14:07 ` Vladimir Sementsov-Ogievskiy
8 siblings, 1 reply; 25+ messages in thread
From: Mark Cave-Ayland @ 2026-05-27 10:46 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, jasowang, mst
Cc: armbru, peterx, farosas, raphael.s.norwitz, bchaney, qemu-devel,
berrange, pbonzini, yc-core
On 22/05/2026 13:05, Vladimir Sementsov-Ogievskiy wrote:
> Hi all!
>
> Here is a new migration parameter "local", which allows to
> enable local migration of TAP virtio-net backend (and maybe other
> devices and backends in future), including its properties and open
> fds.
>
> With this new option, management software doesn't need to initialize
> new TAP and do a switch to it. Nothing should be done around
> virtio-net in local migration: it just migrates and continues to use
> same TAP device. So we avoid extra logic in management software, extra
> allocations in kernel (for new TAP), and corresponding extra delay in
> migration downtime.
>
> v16: rebase on master (on top of "[PATCH v4 00/13] net: refactoring and fixes"),
> 01-08: add r-b by Ben
> 06: add a-b by Markus
> 07: small fix, exclude vnet_hdr= and ifname= arguments when
> incoming-fds is set (they are not allowed actually, and
> netdev_add fails), keep r-b by Ben.
>
> Based-on: <20260318113144.15697-1-vsementsov@yandex-team.ru>
> "[PATCH v4 00/13] net: refactoring and fixes"
>
> Vladimir Sementsov-Ogievskiy (8):
> net/tap: move vhost-net open() calls to tap_parse_vhost_fds()
> net/tap: move vhost initialization to tap_setup_vhost()
> qapi: add local migration parameter
> net: introduce vmstate_net_peer_backend
> virtio-net: support local migration of backend
> net/tap: support local migration with virtio-net
> tests/functional: add skipWithoutSudo() decorator
> tests/functional: add test_tap_migration
>
> hw/core/machine.c | 1 +
> hw/i386/pc_q35.c | 1 +
> hw/net/virtio-net.c | 137 +++++-
> include/hw/virtio/virtio-net.h | 2 +
> include/migration/misc.h | 2 +
> include/net/net.h | 6 +
> migration/options.c | 18 +-
> net/net.c | 47 ++
> net/tap.c | 246 ++++++++--
> qapi/migration.json | 12 +-
> qapi/net.json | 10 +-
> tests/functional/qemu_test/decorators.py | 16 +
> tests/functional/x86_64/meson.build | 1 +
> tests/functional/x86_64/test_tap_migration.py | 458 ++++++++++++++++++
> 14 files changed, 906 insertions(+), 51 deletions(-)
> create mode 100755 tests/functional/x86_64/test_tap_migration.py
I've been looking at this series to try and understand how CPR migration
works with network devices, so I'd be interested to have a look at this.
Do you have a working example command line and monitor sequence as a
reference? I've tried a few examples but I appear to be missing some
context, and there's nothing in the cover letter to help out :)
Also do you have a git branch somewhere with your latest changes?
ATB,
Mark.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v16 5/8] virtio-net: support local migration of backend
2026-05-26 11:23 ` Vladimir Sementsov-Ogievskiy
2026-05-26 17:47 ` Peter Xu
@ 2026-05-27 11:35 ` Vladimir Sementsov-Ogievskiy
2026-05-27 17:19 ` Vladimir Sementsov-Ogievskiy
1 sibling, 1 reply; 25+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2026-05-27 11:35 UTC (permalink / raw)
To: Peter Xu
Cc: Michael S. Tsirkin, jasowang, armbru, farosas, raphael.s.norwitz,
bchaney, qemu-devel, berrange, pbonzini, yc-core,
Philippe Mathieu-Daudé, Zhao Liu, Richard Henderson
On 26.05.26 14:23, Vladimir Sementsov-Ogievskiy wrote:
> +
> +type_init(tap_register_types)
> diff --git a/qapi/net.json b/qapi/net.json
> index 82ddbb51cd7..029f5a4532c 100644
> --- a/qapi/net.json
> +++ b/qapi/net.json
> @@ -429,9 +429,18 @@
a bit more context:
# @incoming-fds: do not open or create any TAP devices. Prepare for
> # getting TAP file descriptors from incoming migration stream.
> # The option is incompatible with any of @fd, @fds, @helper, @br,
> # @ifname, @sndbuf and @vnet_hdr options, and requires @script and
> -# @downscript be explicitly set to nothing (empty string or "no")
> +# @downscript be explicitly set to nothing (empty string or "no"),
> +# and requires also @local-migration to be set an "local"
> +# migration parameter be set as well.
> # (Since 11.1)
> #
> +# @local-migration: enable local migration for this TAP backend.
> +# When set, local migration is enabled/disabled by "local"
> +# migration parameter for this TAP backend. When unset, "local"
> +# migration parameter is ignored for this TAP backend.
> +# (Since 11.1. Defaults to true for MT >= 11.1,
> +# and to false for MT < 11.1)
> +#
> # Since: 1.2
> ##
> { 'struct': 'NetdevTapOptions',
> @@ -451,7 +460,8 @@
> '*vhostforce': 'bool',
> '*queues': 'uint32',
> '*poll-us': 'uint32',
> - '*incoming-fds': 'bool' } }
> + '*incoming-fds': 'bool',
> + '*local-migration': 'bool' } }
Having both "incoming-fds" and "local-migration" (or rename
it to "support-local-migration") seems too much.
But, we can't simply drop "incoming-fds" and rely only on
"local-migration", a this make logic a lot more complicated,
as "local" migration parameter starts to "decide" how tap
initialization works:
We have to postpone opening any files up to "pre-incoming"
point, when all migration parameters are known, to decide:
if "local-migration"=true and "local"=true - do not open
any files, wait for incoming FDs
if "local-migration"=true and "local"-false - open files
In this picture, the fact that migration parameter influence
device initialization - seems rather bad precedent, which is
better to avoid.
On the other hand, with "incoming-fds" parameter, everything is
explicit. We can simply detect and error-out incorrect
combinations (like incoming-fds=true, but incoming migration
started with local=false, etc.), but user is sure, that target
QEMU process will not open any files with this parameter set.
That's a safe way.
Still, I have alternative idea:
Instead of combination "-incoming defer" + "incoming-fds=true",
make a new "-incoming local".
"-incoming local" would be equal to "-incoming defer", but also
implies local=true (i.e. starting incoming migration with
local=false will simply fail).
This way, we know that local=True from the start of the process,
and don't have to wait for some pre-incoming point to be sure
in value of "local" parameter.
So, we simply check "local-migration == True + incoming == local"
instead of "incoming-fds == True" in tap code, and we are done.
What do you think?
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v16 0/8] virtio-net: live-TAP local migration
2026-05-27 10:46 ` [PATCH v16 0/8] virtio-net: live-TAP local migration Mark Cave-Ayland
@ 2026-05-27 14:07 ` Vladimir Sementsov-Ogievskiy
2026-05-27 14:11 ` Vladimir Sementsov-Ogievskiy
0 siblings, 1 reply; 25+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2026-05-27 14:07 UTC (permalink / raw)
To: Mark Cave-Ayland, jasowang, mst
Cc: armbru, peterx, farosas, raphael.s.norwitz, bchaney, qemu-devel,
berrange, pbonzini, yc-core
On 27.05.26 13:46, Mark Cave-Ayland wrote:
> On 22/05/2026 13:05, Vladimir Sementsov-Ogievskiy wrote:
>
>> Hi all!
>>
>> Here is a new migration parameter "local", which allows to
>> enable local migration of TAP virtio-net backend (and maybe other
>> devices and backends in future), including its properties and open
>> fds.
>>
>> With this new option, management software doesn't need to initialize
>> new TAP and do a switch to it. Nothing should be done around
>> virtio-net in local migration: it just migrates and continues to use
>> same TAP device. So we avoid extra logic in management software, extra
>> allocations in kernel (for new TAP), and corresponding extra delay in
>> migration downtime.
>>
>> v16: rebase on master (on top of "[PATCH v4 00/13] net: refactoring and fixes"),
>> 01-08: add r-b by Ben
>> 06: add a-b by Markus
>> 07: small fix, exclude vnet_hdr= and ifname= arguments when
>> incoming-fds is set (they are not allowed actually, and
>> netdev_add fails), keep r-b by Ben.
>>
>> Based-on: <20260318113144.15697-1-vsementsov@yandex-team.ru>
>> "[PATCH v4 00/13] net: refactoring and fixes"
>>
>> Vladimir Sementsov-Ogievskiy (8):
>> net/tap: move vhost-net open() calls to tap_parse_vhost_fds()
>> net/tap: move vhost initialization to tap_setup_vhost()
>> qapi: add local migration parameter
>> net: introduce vmstate_net_peer_backend
>> virtio-net: support local migration of backend
>> net/tap: support local migration with virtio-net
>> tests/functional: add skipWithoutSudo() decorator
>> tests/functional: add test_tap_migration
>>
>> hw/core/machine.c | 1 +
>> hw/i386/pc_q35.c | 1 +
>> hw/net/virtio-net.c | 137 +++++-
>> include/hw/virtio/virtio-net.h | 2 +
>> include/migration/misc.h | 2 +
>> include/net/net.h | 6 +
>> migration/options.c | 18 +-
>> net/net.c | 47 ++
>> net/tap.c | 246 ++++++++--
>> qapi/migration.json | 12 +-
>> qapi/net.json | 10 +-
>> tests/functional/qemu_test/decorators.py | 16 +
>> tests/functional/x86_64/meson.build | 1 +
>> tests/functional/x86_64/test_tap_migration.py | 458 ++++++++++++++++++
>> 14 files changed, 906 insertions(+), 51 deletions(-)
>> create mode 100755 tests/functional/x86_64/test_tap_migration.py
>
> I've been looking at this series to try and understand how CPR migration works with network devices, so I'd be interested to have a look at this.
>
> Do you have a working example command line and monitor sequence as a reference? I've tried a few examples but I appear to be missing some context, and there's nothing in the cover letter to help out :)
>
> Also do you have a git branch somewhere with your latest changes?
>
>
First, the interface is still under discussion (under 5/8 patch of this series).
Still, of course, you may experiment with current version, I've pushed it now to
https://gitlab.com/vsementsov/qemu.git , tag: up-tap-fd-migration-with-bk-opt-v16
There is a test, you may run it, like this:
sudo PYTHONPATH=python:tests/functional QEMU_TEST_QEMU_BINARY=$PWD/build/qemu-system-x86_64 QEMU_TEST_KEEP_SCRATCH=1 MESON_BUILD_ROOT=$PWD/build ./build/pyvenv/bin/python3 tests/functional/x86_64/ test_tap_migration.py
sudo is needed, as test configures TAP devices.
In short, the interface is as follows:
1. Set migration parameter local=True, both on source and target
2. Set virtio-net field local-migration=True, both on source and target
3. Set tap parameter incoming-fds=True on target
4. Start migration
Also, migration channel must be a UNIX socket.
[You see, nothing related to CPR, but it should be possible to use this
tap-fd-migration together with CPR, but I didn't test it]
And longer instruction, to partly reproduce test_tap_fd_migration() by hand:
1. Prerequisites
----------------
QEMU=/path/to/your/build/qemu-system-x86_64
# download same image as in test
wget -O /tmp/alpine.iso "https://dl-cdn.alpinelinux.org/alpine/v3.22/releases/x86_64/alpine-standard-3.22.1-x86_64.iso"
# prepare tap device (be careful to not break your own networks)
sudo ip tuntap add dev tap0 mode tap multi_queue
sudo ip addr add 192.168.100.1/24 dev tap0
sudo ip link set tap0 up
2. Start source VM
------------------
Open terminal A, and run:
$QEMU \
-name source \
-machine q35 \
-accel kvm \
-m 1G \
-object memory-backend-file,id=ram0,size=1G,mem-path=/dev/shm/qemu_migration_test,share=on \
-machine memory-backend=ram0 \
-drive file=/tmp/alpine.iso,media=cdrom,format=raw \
-device pcie-pci-bridge,id=pci.1,bus=pcie.0 \
-netdev tap,id=netdev.1,ifname=tap0,queues=4,vnet_hdr=on,script=no,downscript=no,local-migration=on \
-device virtio-net-pci,netdev=netdev.1,id=vnet.1,bus=pci.1,mq=on,vectors=18,romfile=,disable-legacy=off \
-serial stdio \
-nographic \
-qmp unix:/tmp/qmp-source.sock,server=on,wait=off
Wait for Alpine to boot. When you see the login prompt, log in as root
(no password):
localhost login: root
Configure the guest network:
ip addr add 192.168.100.2/24 dev eth0
ip link set eth0 up
Verify connectivity from the guest:
ping -c 3 192.168.100.1
And from the host (in another terminal):
ping -c 3 192.168.100.2
3. Start target VM
------------------
Open terminal B. The target VM uses the same shared memory file and
the same ISO. The NIC is configured with incoming-fds=on — it will
not open tap0 itself; instead it will receive the TAP file descriptors
from the source over the migration channel.
$QEMU \
-name target \
-machine q35 \
-accel kvm \
-m 1G \
-object memory-backend-file,id=ram0,size=1G,mem-path=/dev/shm/qemu_migration_test,share=on \
-machine memory-backend=ram0 \
-drive file=/tmp/alpine.iso,media=cdrom,format=raw \
-device pcie-pci-bridge,id=pci.1,bus=pcie.0 \
-netdev tap,id=netdev.1,queues=4,script=no,downscript=no,local-migration=on,incoming-fds=on \
-device virtio-net-pci,netdev=netdev.1,id=vnet.1,bus=pci.1,mq=on,vectors=18,romfile=,disable-legacy=off \
-serial stdio \
-nographic \
-qmp unix:/tmp/qmp-target.sock,server=on,wait=off \
-incoming defer
4. Start migration
------------------
Open terminal C and connect to the source QMP socket:
socat - UNIX-CONNECT:/tmp/qmp-source.sock
Negotiate capabilities and configure migration:
{"execute": "qmp_capabilities"}
{"execute": "migrate-set-capabilities", "arguments": {
"capabilities": [
{"capability": "events", "state": true},
{"capability": "x-ignore-shared", "state": true}
]
}}
{"execute": "migrate-set-parameters", "arguments": {"local": true}}
Open terminal D and connect to the target QMP socket:
socat - UNIX-CONNECT:/tmp/qmp-target.sock
Negotiate capabilities and configure migration:
{"execute": "qmp_capabilities"}
{"execute": "migrate-set-capabilities", "arguments": {
"capabilities": [
{"capability": "events", "state": true},
{"capability": "x-ignore-shared", "state": true}
]
}}
{"execute": "migrate-set-parameters", "arguments": {"local": true}}
Tell the target to listen for the incoming migration (terminal D):
{"execute": "migrate-incoming",
"arguments": {"uri": "unix:/tmp/migration.sock"}}
Trigger the migration from the source (terminal C):
{"execute": "migrate",
"arguments": {"uri": "unix:/tmp/migration.sock"}}
Poll migration status until it completes (terminal C):
{"execute": "query-migrate"}
# repeat until "status" == "completed"
Or just wait for the MIGRATION event that QEMU emits automatically:
# {"event": "MIGRATION", "data": {"status": "completed"}, ...}
Once the source reports "completed", resume the target VM (terminal D):
{"execute": "cont"}
The target VM is now running with the migrated state and the TAP file
descriptors that were passed from the source.
Verify that the guest is still reachable from the host:
ping -c 3 192.168.100.2
And from inside the guest (terminal B, target serial console):
ping -c 3 192.168.100.1
5. Cleanup
----------
Shut down the target VM (terminal D):
{"execute": "quit"}
Shut down the source VM (terminal C):
{"execute": "quit"}
Remove the TAP device:
sudo ip tuntap del tap0 mode tap multi_queue
Remove the shared memory file:
rm /dev/shm/qemu_migration_test
Remove leftover sockets if they still exist:
rm -f /tmp/migration.sock /tmp/qmp-source.sock /tmp/qmp-target.sock
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v16 0/8] virtio-net: live-TAP local migration
2026-05-27 14:07 ` Vladimir Sementsov-Ogievskiy
@ 2026-05-27 14:11 ` Vladimir Sementsov-Ogievskiy
0 siblings, 0 replies; 25+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2026-05-27 14:11 UTC (permalink / raw)
To: Mark Cave-Ayland, jasowang, mst
Cc: armbru, peterx, farosas, raphael.s.norwitz, bchaney, qemu-devel,
berrange, pbonzini, yc-core
On 27.05.26 17:07, Vladimir Sementsov-Ogievskiy wrote:
> On 27.05.26 13:46, Mark Cave-Ayland wrote:
>> On 22/05/2026 13:05, Vladimir Sementsov-Ogievskiy wrote:
>>
>>> Hi all!
>>>
>>> Here is a new migration parameter "local", which allows to
>>> enable local migration of TAP virtio-net backend (and maybe other
>>> devices and backends in future), including its properties and open
>>> fds.
>>>
>>> With this new option, management software doesn't need to initialize
>>> new TAP and do a switch to it. Nothing should be done around
>>> virtio-net in local migration: it just migrates and continues to use
>>> same TAP device. So we avoid extra logic in management software, extra
>>> allocations in kernel (for new TAP), and corresponding extra delay in
>>> migration downtime.
>>>
>>> v16: rebase on master (on top of "[PATCH v4 00/13] net: refactoring and fixes"),
>>> 01-08: add r-b by Ben
>>> 06: add a-b by Markus
>>> 07: small fix, exclude vnet_hdr= and ifname= arguments when
>>> incoming-fds is set (they are not allowed actually, and
>>> netdev_add fails), keep r-b by Ben.
>>>
>>> Based-on: <20260318113144.15697-1-vsementsov@yandex-team.ru>
>>> "[PATCH v4 00/13] net: refactoring and fixes"
>>>
>>> Vladimir Sementsov-Ogievskiy (8):
>>> net/tap: move vhost-net open() calls to tap_parse_vhost_fds()
>>> net/tap: move vhost initialization to tap_setup_vhost()
>>> qapi: add local migration parameter
>>> net: introduce vmstate_net_peer_backend
>>> virtio-net: support local migration of backend
>>> net/tap: support local migration with virtio-net
>>> tests/functional: add skipWithoutSudo() decorator
>>> tests/functional: add test_tap_migration
>>>
>>> hw/core/machine.c | 1 +
>>> hw/i386/pc_q35.c | 1 +
>>> hw/net/virtio-net.c | 137 +++++-
>>> include/hw/virtio/virtio-net.h | 2 +
>>> include/migration/misc.h | 2 +
>>> include/net/net.h | 6 +
>>> migration/options.c | 18 +-
>>> net/net.c | 47 ++
>>> net/tap.c | 246 ++++++++--
>>> qapi/migration.json | 12 +-
>>> qapi/net.json | 10 +-
>>> tests/functional/qemu_test/decorators.py | 16 +
>>> tests/functional/x86_64/meson.build | 1 +
>>> tests/functional/x86_64/test_tap_migration.py | 458 ++++++++++++++++++
>>> 14 files changed, 906 insertions(+), 51 deletions(-)
>>> create mode 100755 tests/functional/x86_64/test_tap_migration.py
>>
>> I've been looking at this series to try and understand how CPR migration works with network devices, so I'd be interested to have a look at this.
>>
>> Do you have a working example command line and monitor sequence as a reference? I've tried a few examples but I appear to be missing some context, and there's nothing in the cover letter to help out :)
>>
>> Also do you have a git branch somewhere with your latest changes?
>>
>>
>
> First, the interface is still under discussion (under 5/8 patch of this series).
>
> Still, of course, you may experiment with current version, I've pushed it now to
>
> https://gitlab.com/vsementsov/qemu.git , tag: up-tap-fd-migration-with-bk-opt-v16
>
> There is a test, you may run it, like this:
>
> sudo PYTHONPATH=python:tests/functional QEMU_TEST_QEMU_BINARY=$PWD/build/qemu-system-x86_64 QEMU_TEST_KEEP_SCRATCH=1 MESON_BUILD_ROOT=$PWD/build ./build/pyvenv/bin/python3 tests/functional/x86_64/ test_tap_migration.py
>
> sudo is needed, as test configures TAP devices.
>
> In short, the interface is as follows:
>
> 1. Set migration parameter local=True, both on source and target
> 2. Set virtio-net field local-migration=True, both on source and target
> 3. Set tap parameter incoming-fds=True on target
> 4. Start migration
>
> Also, migration channel must be a UNIX socket.
>
> [You see, nothing related to CPR, but it should be possible to use this
> tap-fd-migration together with CPR, but I didn't test it]
>
>
> And longer instruction, to partly reproduce test_tap_fd_migration() by hand:
[..]
> 2. Start source VM
> ------------------
>
> Open terminal A, and run:
>
> $QEMU \
> -name source \
> -machine q35 \
> -accel kvm \
> -m 1G \
> -object memory-backend-file,id=ram0,size=1G,mem-path=/dev/shm/qemu_migration_test,share=on \
> -machine memory-backend=ram0 \
> -drive file=/tmp/alpine.iso,media=cdrom,format=raw \
> -device pcie-pci-bridge,id=pci.1,bus=pcie.0 \
> -netdev tap,id=netdev.1,ifname=tap0,queues=4,vnet_hdr=on,script=no,downscript=no,local-migration=on \
Oops. For v16, local-migration=on should be in virtio-net-pci, not in tap, and save for target process.
> -device virtio-net-pci,netdev=netdev.1,id=vnet.1,bus=pci.1,mq=on,vectors=18,romfile=,disable-legacy=off \
> -serial stdio \
> -nographic \
> -qmp unix:/tmp/qmp-source.sock,server=on,wait=off
>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v16 5/8] virtio-net: support local migration of backend
2026-05-27 11:35 ` Vladimir Sementsov-Ogievskiy
@ 2026-05-27 17:19 ` Vladimir Sementsov-Ogievskiy
2026-05-27 19:41 ` Peter Xu
0 siblings, 1 reply; 25+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2026-05-27 17:19 UTC (permalink / raw)
To: Peter Xu
Cc: Michael S. Tsirkin, jasowang, armbru, farosas, raphael.s.norwitz,
bchaney, qemu-devel, berrange, pbonzini, yc-core,
Philippe Mathieu-Daudé, Zhao Liu, Richard Henderson
On 27.05.26 14:35, Vladimir Sementsov-Ogievskiy wrote:
> On 26.05.26 14:23, Vladimir Sementsov-Ogievskiy wrote:
>> +
>> +type_init(tap_register_types)
>> diff --git a/qapi/net.json b/qapi/net.json
>> index 82ddbb51cd7..029f5a4532c 100644
>> --- a/qapi/net.json
>> +++ b/qapi/net.json
>> @@ -429,9 +429,18 @@
>
> a bit more context:
>
> # @incoming-fds: do not open or create any TAP devices. Prepare for
>
>
>> # getting TAP file descriptors from incoming migration stream.
>> # The option is incompatible with any of @fd, @fds, @helper, @br,
>> # @ifname, @sndbuf and @vnet_hdr options, and requires @script and
>> -# @downscript be explicitly set to nothing (empty string or "no")
>> +# @downscript be explicitly set to nothing (empty string or "no"),
>> +# and requires also @local-migration to be set an "local"
>> +# migration parameter be set as well.
>> # (Since 11.1)
>> #
>> +# @local-migration: enable local migration for this TAP backend.
>> +# When set, local migration is enabled/disabled by "local"
>> +# migration parameter for this TAP backend. When unset, "local"
>> +# migration parameter is ignored for this TAP backend.
>> +# (Since 11.1. Defaults to true for MT >= 11.1,
>> +# and to false for MT < 11.1)
>> +#
>> # Since: 1.2
>> ##
>> { 'struct': 'NetdevTapOptions',
>> @@ -451,7 +460,8 @@
>> '*vhostforce': 'bool',
>> '*queues': 'uint32',
>> '*poll-us': 'uint32',
>> - '*incoming-fds': 'bool' } }
>> + '*incoming-fds': 'bool',
>> + '*local-migration': 'bool' } }
>
>
> Having both "incoming-fds" and "local-migration" (or rename
> it to "support-local-migration") seems too much.
>
> But, we can't simply drop "incoming-fds" and rely only on
> "local-migration", a this make logic a lot more complicated,
> as "local" migration parameter starts to "decide" how tap
> initialization works:
>
> We have to postpone opening any files up to "pre-incoming"
> point, when all migration parameters are known, to decide:
>
> if "local-migration"=true and "local"=true - do not open
> any files, wait for incoming FDs
>
> if "local-migration"=true and "local"-false - open files
>
> In this picture, the fact that migration parameter influence
> device initialization - seems rather bad precedent, which is
> better to avoid.
>
> On the other hand, with "incoming-fds" parameter, everything is
> explicit. We can simply detect and error-out incorrect
> combinations (like incoming-fds=true, but incoming migration
> started with local=false, etc.), but user is sure, that target
> QEMU process will not open any files with this parameter set.
> That's a safe way.
>
>
> Still, I have alternative idea:
>
> Instead of combination "-incoming defer" + "incoming-fds=true",
> make a new "-incoming local".
>
> "-incoming local" would be equal to "-incoming defer", but also
> implies local=true (i.e. starting incoming migration with
> local=false will simply fail).
>
> This way, we know that local=True from the start of the process,
> and don't have to wait for some pre-incoming point to be sure
> in value of "local" parameter.
>
> So, we simply check "local-migration == True + incoming == local"
> instead of "incoming-fds == True" in tap code, and we are done.
>
> What do you think?
>
>
Hmm. Bad idea. It may have same problem if use commandline options
to create devices, as ordering of "-incoming local" and "-netdev"
may affect how tap device will be initialized. It can be solved (or,
maybe, it works as expected even now, if "-incoming" always handled
before "-netdev" options... But anyway that's much more shaky than
simple incoming-fds=True.
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v16 5/8] virtio-net: support local migration of backend
2026-05-27 17:19 ` Vladimir Sementsov-Ogievskiy
@ 2026-05-27 19:41 ` Peter Xu
2026-05-27 20:32 ` Vladimir Sementsov-Ogievskiy
2026-05-27 20:38 ` Vladimir Sementsov-Ogievskiy
0 siblings, 2 replies; 25+ messages in thread
From: Peter Xu @ 2026-05-27 19:41 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy
Cc: Michael S. Tsirkin, jasowang, armbru, farosas, raphael.s.norwitz,
bchaney, qemu-devel, berrange, pbonzini, yc-core,
Philippe Mathieu-Daudé, Zhao Liu, Richard Henderson
On Wed, May 27, 2026 at 08:19:53PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> On 27.05.26 14:35, Vladimir Sementsov-Ogievskiy wrote:
> > On 26.05.26 14:23, Vladimir Sementsov-Ogievskiy wrote:
> > > +
> > > +type_init(tap_register_types)
> > > diff --git a/qapi/net.json b/qapi/net.json
> > > index 82ddbb51cd7..029f5a4532c 100644
> > > --- a/qapi/net.json
> > > +++ b/qapi/net.json
> > > @@ -429,9 +429,18 @@
> >
> > a bit more context:
> >
> > # @incoming-fds: do not open or create any TAP devices. Prepare for
> >
> >
> > > # getting TAP file descriptors from incoming migration stream.
> > > # The option is incompatible with any of @fd, @fds, @helper, @br,
> > > # @ifname, @sndbuf and @vnet_hdr options, and requires @script and
> > > -# @downscript be explicitly set to nothing (empty string or "no")
> > > +# @downscript be explicitly set to nothing (empty string or "no"),
> > > +# and requires also @local-migration to be set an "local"
> > > +# migration parameter be set as well.
> > > # (Since 11.1)
> > > #
> > > +# @local-migration: enable local migration for this TAP backend.
> > > +# When set, local migration is enabled/disabled by "local"
> > > +# migration parameter for this TAP backend. When unset, "local"
> > > +# migration parameter is ignored for this TAP backend.
> > > +# (Since 11.1. Defaults to true for MT >= 11.1,
> > > +# and to false for MT < 11.1)
> > > +#
> > > # Since: 1.2
> > > ##
> > > { 'struct': 'NetdevTapOptions',
> > > @@ -451,7 +460,8 @@
> > > '*vhostforce': 'bool',
> > > '*queues': 'uint32',
> > > '*poll-us': 'uint32',
> > > - '*incoming-fds': 'bool' } }
> > > + '*incoming-fds': 'bool',
> > > + '*local-migration': 'bool' } }
> >
> >
> > Having both "incoming-fds" and "local-migration" (or rename
> > it to "support-local-migration") seems too much.
> >
> > But, we can't simply drop "incoming-fds" and rely only on
> > "local-migration", a this make logic a lot more complicated,
> > as "local" migration parameter starts to "decide" how tap
> > initialization works:
> >
> > We have to postpone opening any files up to "pre-incoming"
> > point, when all migration parameters are known, to decide:
> >
> > if "local-migration"=true and "local"=true - do not open
> > any files, wait for incoming FDs
> >
> > if "local-migration"=true and "local"-false - open files
> >
> > In this picture, the fact that migration parameter influence
> > device initialization - seems rather bad precedent, which is
> > better to avoid.
> >
> > On the other hand, with "incoming-fds" parameter, everything is
> > explicit. We can simply detect and error-out incorrect
> > combinations (like incoming-fds=true, but incoming migration
> > started with local=false, etc.), but user is sure, that target
> > QEMU process will not open any files with this parameter set.
> > That's a safe way.
> >
> >
> > Still, I have alternative idea:
> >
> > Instead of combination "-incoming defer" + "incoming-fds=true",
> > make a new "-incoming local".
> >
> > "-incoming local" would be equal to "-incoming defer", but also
> > implies local=true (i.e. starting incoming migration with
> > local=false will simply fail).
> >
> > This way, we know that local=True from the start of the process,
> > and don't have to wait for some pre-incoming point to be sure
> > in value of "local" parameter.
> >
> > So, we simply check "local-migration == True + incoming == local"
> > instead of "incoming-fds == True" in tap code, and we are done.
> >
> > What do you think?
> >
> >
>
> Hmm. Bad idea. It may have same problem if use commandline options
> to create devices, as ordering of "-incoming local" and "-netdev"
> may affect how tap device will be initialized. It can be solved (or,
> maybe, it works as expected even now, if "-incoming" always handled
> before "-netdev" options... But anyway that's much more shaky than
> simple incoming-fds=True.
Personally I liked part of your proposal.. normally I would rather leave it
for later, but then it means we will be stuck with this incoming-fds API.
So I'll still leave the thoughts below for consideration.
In general, to me this is another requirement to specify migration
parameters during early boot.
We used to tackle with this problem a few times, in many cases by
reordering of initialization of the migration object, which can be error
prone.
We almost moved to another model where we moved some special migration
parameters out of the migration object, into some global variables instead,
then they're available since the start of QEMU.
Two examples I am aware of while looking at this (maybe more):
- The -only-migratable option
This one used to be a "parameter-like" option, but then things broke, we
moved to a global only_migratable variable, to make it available during
early boot.
Currently, it consumes a special option, QEMU_OPTION_only_migratable.
- The "mode" parameter (used by CPR-transfer)
This one is literally a parameter even now, CPR-transfer needs it to be
set even before loading of CPR channel, which is very, very early..
It's done by quite a few complex steps:
- hijacking -incoming parameter, in incoming_option_parse() we first
allow setup of CPR channels,
- then at cpr_state_load(), it consumes that channel when set, invoke
cpr_set_incoming_mode(), which sets a magical global var ( :( ) called
incoming_mode,
- then, further hijack migrate_mode(), which normally should just fetch
"mode" parameter from migration object, peek at incoming_mode first,
when set, overwrite the "mode" parameter...
Now this is the third one: we actually want to have "local" parameter
available.
I wonder if we should just have some migration parameters to be special to
be set early, maintained in a single global place, then when creating
migration objects we should apply these to migration object, making sure
it's consistent.
We could actually reuse -incoming, so far it supprts:
- "defer", as a string
- URI, in form of (SOME_WORD:.*)
- then we assume it's a channel
Since we already have defer, we could make a special case for it, say,
-incoming config:key1=value1,key2=value2,...
With that, migration options can be set at boot and can be referenced
anytime, we don't need to worry on migration object init ordering. We
could obsolete -only-migratable but allow:
-incoming config:only-migrate=on
For CPR, to keep compatibility we still need to set mode=cpr-* silently,
but then we should be able to be able to reference any migration parameters
anytime, including local= now, removing incoming-fds TAP option.
I'm not sure if this is a good approach, but we can think about it. I do
worry we have future demands on similar things, then we need to tackle it
sooner or later (we will want to stop introducing special parameters at
some point, like incoming-fds=on).
Thanks,
--
Peter Xu
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v16 5/8] virtio-net: support local migration of backend
2026-05-27 19:41 ` Peter Xu
@ 2026-05-27 20:32 ` Vladimir Sementsov-Ogievskiy
2026-05-28 13:45 ` Peter Xu
2026-05-27 20:38 ` Vladimir Sementsov-Ogievskiy
1 sibling, 1 reply; 25+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2026-05-27 20:32 UTC (permalink / raw)
To: Peter Xu
Cc: Michael S. Tsirkin, jasowang, armbru, farosas, raphael.s.norwitz,
bchaney, qemu-devel, berrange, pbonzini, yc-core,
Philippe Mathieu-Daudé, Zhao Liu, Richard Henderson
On 27.05.26 22:41, Peter Xu wrote:
> On Wed, May 27, 2026 at 08:19:53PM +0300, Vladimir Sementsov-Ogievskiy wrote:
>> On 27.05.26 14:35, Vladimir Sementsov-Ogievskiy wrote:
>>> On 26.05.26 14:23, Vladimir Sementsov-Ogievskiy wrote:
>>>> +
>>>> +type_init(tap_register_types)
>>>> diff --git a/qapi/net.json b/qapi/net.json
>>>> index 82ddbb51cd7..029f5a4532c 100644
>>>> --- a/qapi/net.json
>>>> +++ b/qapi/net.json
>>>> @@ -429,9 +429,18 @@
>>>
>>> a bit more context:
>>>
>>> # @incoming-fds: do not open or create any TAP devices. Prepare for
>>>
>>>
>>>> # getting TAP file descriptors from incoming migration stream.
>>>> # The option is incompatible with any of @fd, @fds, @helper, @br,
>>>> # @ifname, @sndbuf and @vnet_hdr options, and requires @script and
>>>> -# @downscript be explicitly set to nothing (empty string or "no")
>>>> +# @downscript be explicitly set to nothing (empty string or "no"),
>>>> +# and requires also @local-migration to be set an "local"
>>>> +# migration parameter be set as well.
>>>> # (Since 11.1)
>>>> #
>>>> +# @local-migration: enable local migration for this TAP backend.
>>>> +# When set, local migration is enabled/disabled by "local"
>>>> +# migration parameter for this TAP backend. When unset, "local"
>>>> +# migration parameter is ignored for this TAP backend.
>>>> +# (Since 11.1. Defaults to true for MT >= 11.1,
>>>> +# and to false for MT < 11.1)
>>>> +#
>>>> # Since: 1.2
>>>> ##
>>>> { 'struct': 'NetdevTapOptions',
>>>> @@ -451,7 +460,8 @@
>>>> '*vhostforce': 'bool',
>>>> '*queues': 'uint32',
>>>> '*poll-us': 'uint32',
>>>> - '*incoming-fds': 'bool' } }
>>>> + '*incoming-fds': 'bool',
>>>> + '*local-migration': 'bool' } }
>>>
>>>
>>> Having both "incoming-fds" and "local-migration" (or rename
>>> it to "support-local-migration") seems too much.
>>>
>>> But, we can't simply drop "incoming-fds" and rely only on
>>> "local-migration", a this make logic a lot more complicated,
>>> as "local" migration parameter starts to "decide" how tap
>>> initialization works:
>>>
>>> We have to postpone opening any files up to "pre-incoming"
>>> point, when all migration parameters are known, to decide:
>>>
>>> if "local-migration"=true and "local"=true - do not open
>>> any files, wait for incoming FDs
>>>
>>> if "local-migration"=true and "local"-false - open files
>>>
>>> In this picture, the fact that migration parameter influence
>>> device initialization - seems rather bad precedent, which is
>>> better to avoid.
>>>
>>> On the other hand, with "incoming-fds" parameter, everything is
>>> explicit. We can simply detect and error-out incorrect
>>> combinations (like incoming-fds=true, but incoming migration
>>> started with local=false, etc.), but user is sure, that target
>>> QEMU process will not open any files with this parameter set.
>>> That's a safe way.
>>>
>>>
>>> Still, I have alternative idea:
>>>
>>> Instead of combination "-incoming defer" + "incoming-fds=true",
>>> make a new "-incoming local".
>>>
>>> "-incoming local" would be equal to "-incoming defer", but also
>>> implies local=true (i.e. starting incoming migration with
>>> local=false will simply fail).
>>>
>>> This way, we know that local=True from the start of the process,
>>> and don't have to wait for some pre-incoming point to be sure
>>> in value of "local" parameter.
>>>
>>> So, we simply check "local-migration == True + incoming == local"
>>> instead of "incoming-fds == True" in tap code, and we are done.
>>>
>>> What do you think?
>>>
>>>
>>
>> Hmm. Bad idea. It may have same problem if use commandline options
>> to create devices, as ordering of "-incoming local" and "-netdev"
>> may affect how tap device will be initialized. It can be solved (or,
>> maybe, it works as expected even now, if "-incoming" always handled
>> before "-netdev" options... But anyway that's much more shaky than
>> simple incoming-fds=True.
>
> Personally I liked part of your proposal.. normally I would rather leave it
> for later, but then it means we will be stuck with this incoming-fds API.
> So I'll still leave the thoughts below for consideration.
>
> In general, to me this is another requirement to specify migration
> parameters during early boot.
>
> We used to tackle with this problem a few times, in many cases by
> reordering of initialization of the migration object, which can be error
> prone.
>
> We almost moved to another model where we moved some special migration
> parameters out of the migration object, into some global variables instead,
> then they're available since the start of QEMU.
>
> Two examples I am aware of while looking at this (maybe more):
>
> - The -only-migratable option
>
> This one used to be a "parameter-like" option, but then things broke, we
> moved to a global only_migratable variable, to make it available during
> early boot.
>
> Currently, it consumes a special option, QEMU_OPTION_only_migratable.
>
> - The "mode" parameter (used by CPR-transfer)
>
> This one is literally a parameter even now, CPR-transfer needs it to be
> set even before loading of CPR channel, which is very, very early..
>
> It's done by quite a few complex steps:
>
> - hijacking -incoming parameter, in incoming_option_parse() we first
> allow setup of CPR channels,
>
> - then at cpr_state_load(), it consumes that channel when set, invoke
> cpr_set_incoming_mode(), which sets a magical global var ( :( ) called
> incoming_mode,
>
> - then, further hijack migrate_mode(), which normally should just fetch
> "mode" parameter from migration object, peek at incoming_mode first,
> when set, overwrite the "mode" parameter...
>
> Now this is the third one: we actually want to have "local" parameter
> available.
>
> I wonder if we should just have some migration parameters to be special to
> be set early, maintained in a single global place, then when creating
> migration objects we should apply these to migration object, making sure
> it's consistent.
>
> We could actually reuse -incoming, so far it supprts:
>
> - "defer", as a string
> - URI, in form of (SOME_WORD:.*)
> - then we assume it's a channel
>
> Since we already have defer, we could make a special case for it, say,
>
> -incoming config:key1=value1,key2=value2,...
>
> With that, migration options can be set at boot and can be referenced
> anytime, we don't need to worry on migration object init ordering. We
> could obsolete -only-migratable but allow:
>
> -incoming config:only-migrate=on
>
> For CPR, to keep compatibility we still need to set mode=cpr-* silently,
> but then we should be able to be able to reference any migration parameters
> anytime, including local= now, removing incoming-fds TAP option.
>
> I'm not sure if this is a good approach, but we can think about it. I do
> worry we have future demands on similar things, then we need to tackle it
> sooner or later (we will want to stop introducing special parameters at
> some point, like incoming-fds=on).
>
> Thanks,
>
What make me doubt is asymmetry between outgoing and incoming config:
If we add early "local" parameter, which is set by
"-incoming config:local=on", what about normal "local" parameter?
If we keep it as is, user may think, that config:local=on and setting
local parameter are the same, and chose setting the parameter through
QMP (which is bad for us)..
So, seems, local migration parameter and local early parameter should
be different things.
Maybe, rename "local" parameter to "outgoing-local", and ignore it for
incoming migration. And add "-incoming config:incoming-local=on", which
produces "incoming-local" parameter, which may be set _only_ through
"-incoming" option and cannot be set by migrate-set-parameters, to avoid
any kind of misuse.
Doesn't look ideal :/ And it breaks the rule that migration parameters
should be the same for source and target for migration to work properly.
OK, the workaround is: keep "local" name for both "early" and "normal"
versions of parameter, and add some checks:
- If user tries to set "local" parameter during incoming migration,
but "early local" is not set - fail
- If user tries to run incoming migration, when "early local" is set
but "normal local" is not - fail too
This way, the only possible and correct way to turn on local migration
on target is to set both "early" and "normal" local parameter. Still,
not ideal.
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v16 5/8] virtio-net: support local migration of backend
2026-05-27 19:41 ` Peter Xu
2026-05-27 20:32 ` Vladimir Sementsov-Ogievskiy
@ 2026-05-27 20:38 ` Vladimir Sementsov-Ogievskiy
2026-05-28 13:41 ` Peter Xu
1 sibling, 1 reply; 25+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2026-05-27 20:38 UTC (permalink / raw)
To: Peter Xu
Cc: Michael S. Tsirkin, jasowang, armbru, farosas, raphael.s.norwitz,
bchaney, qemu-devel, berrange, pbonzini, yc-core,
Philippe Mathieu-Daudé, Zhao Liu, Richard Henderson
On 27.05.26 22:41, Peter Xu wrote:
> Personally I liked part of your proposal.. normally I would rather leave it
> for later, but then it means we will be stuck with this incoming-fds API.
I am OK to mark "incoming-fds" as unstable, to simplify future rework,
if there is a good solution.
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v16 5/8] virtio-net: support local migration of backend
2026-05-27 20:38 ` Vladimir Sementsov-Ogievskiy
@ 2026-05-28 13:41 ` Peter Xu
0 siblings, 0 replies; 25+ messages in thread
From: Peter Xu @ 2026-05-28 13:41 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy
Cc: Michael S. Tsirkin, jasowang, armbru, farosas, raphael.s.norwitz,
bchaney, qemu-devel, berrange, pbonzini, yc-core,
Philippe Mathieu-Daudé, Zhao Liu, Richard Henderson
On Wed, May 27, 2026 at 11:38:14PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> On 27.05.26 22:41, Peter Xu wrote:
> > Personally I liked part of your proposal.. normally I would rather leave it
> > for later, but then it means we will be stuck with this incoming-fds API.
>
> I am OK to mark "incoming-fds" as unstable, to simplify future rework,
> if there is a good solution.
Sure, if that's unstable then it's not a concern, I also don't want to
block this series as late. I'll see if I can prepare some RFC patches this
week or next for this as a general problem.
--
Peter Xu
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v16 5/8] virtio-net: support local migration of backend
2026-05-27 20:32 ` Vladimir Sementsov-Ogievskiy
@ 2026-05-28 13:45 ` Peter Xu
0 siblings, 0 replies; 25+ messages in thread
From: Peter Xu @ 2026-05-28 13:45 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy
Cc: Michael S. Tsirkin, jasowang, armbru, farosas, raphael.s.norwitz,
bchaney, qemu-devel, berrange, pbonzini, yc-core,
Philippe Mathieu-Daudé, Zhao Liu, Richard Henderson
On Wed, May 27, 2026 at 11:32:24PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> On 27.05.26 22:41, Peter Xu wrote:
> > On Wed, May 27, 2026 at 08:19:53PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> > > On 27.05.26 14:35, Vladimir Sementsov-Ogievskiy wrote:
> > > > On 26.05.26 14:23, Vladimir Sementsov-Ogievskiy wrote:
> > > > > +
> > > > > +type_init(tap_register_types)
> > > > > diff --git a/qapi/net.json b/qapi/net.json
> > > > > index 82ddbb51cd7..029f5a4532c 100644
> > > > > --- a/qapi/net.json
> > > > > +++ b/qapi/net.json
> > > > > @@ -429,9 +429,18 @@
> > > >
> > > > a bit more context:
> > > >
> > > > # @incoming-fds: do not open or create any TAP devices. Prepare for
> > > >
> > > >
> > > > > # getting TAP file descriptors from incoming migration stream.
> > > > > # The option is incompatible with any of @fd, @fds, @helper, @br,
> > > > > # @ifname, @sndbuf and @vnet_hdr options, and requires @script and
> > > > > -# @downscript be explicitly set to nothing (empty string or "no")
> > > > > +# @downscript be explicitly set to nothing (empty string or "no"),
> > > > > +# and requires also @local-migration to be set an "local"
> > > > > +# migration parameter be set as well.
> > > > > # (Since 11.1)
> > > > > #
> > > > > +# @local-migration: enable local migration for this TAP backend.
> > > > > +# When set, local migration is enabled/disabled by "local"
> > > > > +# migration parameter for this TAP backend. When unset, "local"
> > > > > +# migration parameter is ignored for this TAP backend.
> > > > > +# (Since 11.1. Defaults to true for MT >= 11.1,
> > > > > +# and to false for MT < 11.1)
> > > > > +#
> > > > > # Since: 1.2
> > > > > ##
> > > > > { 'struct': 'NetdevTapOptions',
> > > > > @@ -451,7 +460,8 @@
> > > > > '*vhostforce': 'bool',
> > > > > '*queues': 'uint32',
> > > > > '*poll-us': 'uint32',
> > > > > - '*incoming-fds': 'bool' } }
> > > > > + '*incoming-fds': 'bool',
> > > > > + '*local-migration': 'bool' } }
> > > >
> > > >
> > > > Having both "incoming-fds" and "local-migration" (or rename
> > > > it to "support-local-migration") seems too much.
> > > >
> > > > But, we can't simply drop "incoming-fds" and rely only on
> > > > "local-migration", a this make logic a lot more complicated,
> > > > as "local" migration parameter starts to "decide" how tap
> > > > initialization works:
> > > >
> > > > We have to postpone opening any files up to "pre-incoming"
> > > > point, when all migration parameters are known, to decide:
> > > >
> > > > if "local-migration"=true and "local"=true - do not open
> > > > any files, wait for incoming FDs
> > > >
> > > > if "local-migration"=true and "local"-false - open files
> > > >
> > > > In this picture, the fact that migration parameter influence
> > > > device initialization - seems rather bad precedent, which is
> > > > better to avoid.
> > > >
> > > > On the other hand, with "incoming-fds" parameter, everything is
> > > > explicit. We can simply detect and error-out incorrect
> > > > combinations (like incoming-fds=true, but incoming migration
> > > > started with local=false, etc.), but user is sure, that target
> > > > QEMU process will not open any files with this parameter set.
> > > > That's a safe way.
> > > >
> > > >
> > > > Still, I have alternative idea:
> > > >
> > > > Instead of combination "-incoming defer" + "incoming-fds=true",
> > > > make a new "-incoming local".
> > > >
> > > > "-incoming local" would be equal to "-incoming defer", but also
> > > > implies local=true (i.e. starting incoming migration with
> > > > local=false will simply fail).
> > > >
> > > > This way, we know that local=True from the start of the process,
> > > > and don't have to wait for some pre-incoming point to be sure
> > > > in value of "local" parameter.
> > > >
> > > > So, we simply check "local-migration == True + incoming == local"
> > > > instead of "incoming-fds == True" in tap code, and we are done.
> > > >
> > > > What do you think?
> > > >
> > > >
> > >
> > > Hmm. Bad idea. It may have same problem if use commandline options
> > > to create devices, as ordering of "-incoming local" and "-netdev"
> > > may affect how tap device will be initialized. It can be solved (or,
> > > maybe, it works as expected even now, if "-incoming" always handled
> > > before "-netdev" options... But anyway that's much more shaky than
> > > simple incoming-fds=True.
> >
> > Personally I liked part of your proposal.. normally I would rather leave it
> > for later, but then it means we will be stuck with this incoming-fds API.
> > So I'll still leave the thoughts below for consideration.
> >
> > In general, to me this is another requirement to specify migration
> > parameters during early boot.
> >
> > We used to tackle with this problem a few times, in many cases by
> > reordering of initialization of the migration object, which can be error
> > prone.
> >
> > We almost moved to another model where we moved some special migration
> > parameters out of the migration object, into some global variables instead,
> > then they're available since the start of QEMU.
> >
> > Two examples I am aware of while looking at this (maybe more):
> >
> > - The -only-migratable option
> >
> > This one used to be a "parameter-like" option, but then things broke, we
> > moved to a global only_migratable variable, to make it available during
> > early boot.
> >
> > Currently, it consumes a special option, QEMU_OPTION_only_migratable.
> >
> > - The "mode" parameter (used by CPR-transfer)
> >
> > This one is literally a parameter even now, CPR-transfer needs it to be
> > set even before loading of CPR channel, which is very, very early..
> >
> > It's done by quite a few complex steps:
> >
> > - hijacking -incoming parameter, in incoming_option_parse() we first
> > allow setup of CPR channels,
> >
> > - then at cpr_state_load(), it consumes that channel when set, invoke
> > cpr_set_incoming_mode(), which sets a magical global var ( :( ) called
> > incoming_mode,
> >
> > - then, further hijack migrate_mode(), which normally should just fetch
> > "mode" parameter from migration object, peek at incoming_mode first,
> > when set, overwrite the "mode" parameter...
> >
> > Now this is the third one: we actually want to have "local" parameter
> > available.
> >
> > I wonder if we should just have some migration parameters to be special to
> > be set early, maintained in a single global place, then when creating
> > migration objects we should apply these to migration object, making sure
> > it's consistent.
> >
> > We could actually reuse -incoming, so far it supprts:
> >
> > - "defer", as a string
> > - URI, in form of (SOME_WORD:.*)
> > - then we assume it's a channel
> >
> > Since we already have defer, we could make a special case for it, say,
> > -incoming config:key1=value1,key2=value2,...
> >
> > With that, migration options can be set at boot and can be referenced
> > anytime, we don't need to worry on migration object init ordering. We
> > could obsolete -only-migratable but allow:
> >
> > -incoming config:only-migrate=on
> >
> > For CPR, to keep compatibility we still need to set mode=cpr-* silently,
> > but then we should be able to be able to reference any migration parameters
> > anytime, including local= now, removing incoming-fds TAP option.
> >
> > I'm not sure if this is a good approach, but we can think about it. I do
> > worry we have future demands on similar things, then we need to tackle it
> > sooner or later (we will want to stop introducing special parameters at
> > some point, like incoming-fds=on).
> >
> > Thanks,
> >
>
> What make me doubt is asymmetry between outgoing and incoming config:
>
> If we add early "local" parameter, which is set by
> "-incoming config:local=on", what about normal "local" parameter?
When creating the migration object, we should apply all those config:* to
it to keep it consistent. Then the global can be released.
>
> If we keep it as is, user may think, that config:local=on and setting
> local parameter are the same, and chose setting the parameter through
> QMP (which is bad for us)..
Yeah, in documents we need to mark -incoming config:* to be required for
use cases. We can also mention that in the manual about the difference,
but I agree it's not easy to catch.
>
> So, seems, local migration parameter and local early parameter should
> be different things.
>
> Maybe, rename "local" parameter to "outgoing-local", and ignore it for
> incoming migration. And add "-incoming config:incoming-local=on", which
> produces "incoming-local" parameter, which may be set _only_ through
> "-incoming" option and cannot be set by migrate-set-parameters, to avoid
> any kind of misuse.
>
> Doesn't look ideal :/ And it breaks the rule that migration parameters
> should be the same for source and target for migration to work properly.
>
>
> OK, the workaround is: keep "local" name for both "early" and "normal"
> versions of parameter, and add some checks:
>
> - If user tries to set "local" parameter during incoming migration,
> but "early local" is not set - fail
>
> - If user tries to run incoming migration, when "early local" is set
> but "normal local" is not - fail too
>
> This way, the only possible and correct way to turn on local migration
> on target is to set both "early" and "normal" local parameter. Still,
> not ideal.
My goal is to make it really the same thing, no need to two "local"
parameters, no need to differenciate src/dst "local" parameters. They
should be the same. Just to provide a slightly cleaner way to let us not
worry about "whether migration object is created, and what's the order of
creation" kind of problems.
--
Peter Xu
^ permalink raw reply [flat|nested] 25+ messages in thread
end of thread, other threads:[~2026-05-28 13:45 UTC | newest]
Thread overview: 25+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-05-22 12:05 [PATCH v16 0/8] virtio-net: live-TAP local migration Vladimir Sementsov-Ogievskiy
2026-05-22 12:05 ` [PATCH v16 1/8] net/tap: move vhost-net open() calls to tap_parse_vhost_fds() Vladimir Sementsov-Ogievskiy
2026-05-22 12:05 ` [PATCH v16 2/8] net/tap: move vhost initialization to tap_setup_vhost() Vladimir Sementsov-Ogievskiy
2026-05-22 12:05 ` [PATCH v16 3/8] qapi: add local migration parameter Vladimir Sementsov-Ogievskiy
2026-05-22 12:05 ` [PATCH v16 4/8] net: introduce vmstate_net_peer_backend Vladimir Sementsov-Ogievskiy
2026-05-22 12:05 ` [PATCH v16 5/8] virtio-net: support local migration of backend Vladimir Sementsov-Ogievskiy
2026-05-24 9:09 ` Michael S. Tsirkin
2026-05-25 13:51 ` Vladimir Sementsov-Ogievskiy
2026-05-25 14:45 ` Peter Xu
2026-05-26 11:23 ` Vladimir Sementsov-Ogievskiy
2026-05-26 17:47 ` Peter Xu
2026-05-27 7:24 ` Vladimir Sementsov-Ogievskiy
2026-05-27 11:35 ` Vladimir Sementsov-Ogievskiy
2026-05-27 17:19 ` Vladimir Sementsov-Ogievskiy
2026-05-27 19:41 ` Peter Xu
2026-05-27 20:32 ` Vladimir Sementsov-Ogievskiy
2026-05-28 13:45 ` Peter Xu
2026-05-27 20:38 ` Vladimir Sementsov-Ogievskiy
2026-05-28 13:41 ` Peter Xu
2026-05-22 12:05 ` [PATCH v16 6/8] net/tap: support local migration with virtio-net Vladimir Sementsov-Ogievskiy
2026-05-22 12:05 ` [PATCH v16 7/8] tests/functional: add skipWithoutSudo() decorator Vladimir Sementsov-Ogievskiy
2026-05-22 12:05 ` [PATCH v16 8/8] tests/functional: add test_tap_migration Vladimir Sementsov-Ogievskiy
2026-05-27 10:46 ` [PATCH v16 0/8] virtio-net: live-TAP local migration Mark Cave-Ayland
2026-05-27 14:07 ` Vladimir Sementsov-Ogievskiy
2026-05-27 14:11 ` Vladimir Sementsov-Ogievskiy
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.