[RFC V2 0/8] Live update: tap and vhost

[RFC V2 0/8] Live update: tap and vhost

[Steve Sistare]

Tap and vhost devices can be preserved during cpr-transfer using
traditional live migration methods, wherein the management layer
creates new interfaces for the target and fiddles with 'ip link'
to deactivate the old interface and activate the new.

However, CPR can simply send the file descriptors to new QEMU,
with no special management actions required.  The user enables
this behavior by specifing '-netdev tap,cpr=on'.  The default
is cpr=off.

Steve Sistare (8):
  migration: stop vm earlier for cpr
  migration: cpr setup notifier
  vhost: reset vhost devices for cpr
  cpr: delete all fds
  Revert "vhost-backend: remove vhost_kernel_reset_device()"
  tap: common return label
  tap: cpr support
  tap: postload fix for cpr

 qapi/net.json             |   5 +-
 include/hw/virtio/vhost.h |   1 +
 include/migration/cpr.h   |   3 +-
 include/net/tap.h         |   1 +
 hw/net/virtio-net.c       |  20 +++++++
 hw/vfio/device.c          |   2 +-
 hw/virtio/vhost-backend.c |   6 ++
 hw/virtio/vhost.c         |  32 +++++++++++
 migration/cpr.c           |  24 ++++++--
 migration/migration.c     |  38 ++++++++-----
 net/tap-win32.c           |   5 ++
 net/tap.c                 | 141 +++++++++++++++++++++++++++++++++++-----------
 12 files changed, 223 insertions(+), 55 deletions(-)

-- 
1.8.3.1

[RFC V2 1/8] migration: stop vm earlier for cpr

[Steve Sistare]

Stop the vm earlier for cpr, before cpr_save_state which causes new QEMU
to proceed and initialize devices.  We must guarantee devices are stopped
in old QEMU, and all source notifiers called, before they are initialized
in new QEMU.

Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
---
 migration/migration.c | 26 +++++++++++++++++---------
 1 file changed, 17 insertions(+), 9 deletions(-)

diff --git a/migration/migration.c b/migration/migration.c
index 10c216d..2efe60a 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -1662,6 +1662,7 @@ void migration_cancel(void)
                           MIGRATION_STATUS_CANCELLED);
         cpr_state_close();
         migrate_hup_delete(s);
+        vm_resume(s->vm_old_state);
     }
 }
 
@@ -2196,6 +2197,7 @@ void qmp_migrate(const char *uri, bool has_channels,
     MigrationAddress *addr = NULL;
     MigrationChannel *channelv[MIGRATION_CHANNEL_TYPE__MAX] = { NULL };
     MigrationChannel *cpr_channel = NULL;
+    bool stopped = false;
 
     /*
      * Having preliminary checks for uri and channel
@@ -2248,6 +2250,15 @@ void qmp_migrate(const char *uri, bool has_channels,
         return;
     }
 
+    if (migrate_mode_is_cpr(s)) {
+        int ret = migration_stop_vm(s, RUN_STATE_FINISH_MIGRATE);
+        if (ret < 0) {
+            error_setg(&local_err, "migration_stop_vm failed, error %d", -ret);
+            goto out;
+        }
+        stopped = true;
+    }
+
     if (cpr_state_save(cpr_channel, &local_err)) {
         goto out;
     }
@@ -2274,6 +2285,9 @@ out:
     if (local_err) {
         migration_connect_set_error(s, local_err);
         error_propagate(errp, local_err);
+        if (stopped) {
+            vm_resume(s->vm_old_state);
+        }
     }
 }
 
@@ -2319,6 +2333,9 @@ static void qmp_migrate_finish(MigrationAddress *addr, bool resume_requested,
         }
         migration_connect_set_error(s, local_err);
         error_propagate(errp, local_err);
+        if (migrate_mode_is_cpr(s)) {
+            vm_resume(s->vm_old_state);
+        }
         return;
     }
 }
@@ -3961,7 +3978,6 @@ void migration_connect(MigrationState *s, Error *error_in)
     Error *local_err = NULL;
     uint64_t rate_limit;
     bool resume = (s->state == MIGRATION_STATUS_POSTCOPY_RECOVER_SETUP);
-    int ret;
 
     /*
      * If there's a previous error, free it and prepare for another one.
@@ -4033,14 +4049,6 @@ void migration_connect(MigrationState *s, Error *error_in)
         return;
     }
 
-    if (migrate_mode_is_cpr(s)) {
-        ret = migration_stop_vm(s, RUN_STATE_FINISH_MIGRATE);
-        if (ret < 0) {
-            error_setg(&local_err, "migration_stop_vm failed, error %d", -ret);
-            goto fail;
-        }
-    }
-
     /*
      * Take a refcount to make sure the migration object won't get freed by
      * the main thread already in migration_shutdown().
-- 
1.8.3.1

[RFC V2 2/8] migration: cpr setup notifier

[Steve Sistare]

Call MIG_EVENT_PRECOPY_SETUP earlier, before CPR starts.  An early notifier
is needed for resetting vhost devices, as explained in the next patch.

Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
---
 migration/migration.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/migration/migration.c b/migration/migration.c
index 2efe60a..49d1e7d 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -2259,7 +2259,14 @@ void qmp_migrate(const char *uri, bool has_channels,
         stopped = true;
     }
 
+    /* Notify before starting migration thread, and before starting cpr */
+    if (!resume_requested &&
+        migration_call_notifiers(s, MIG_EVENT_PRECOPY_SETUP, &local_err)) {
+        goto out;
+    }
+
     if (cpr_state_save(cpr_channel, &local_err)) {
+        migration_call_notifiers(s, MIG_EVENT_PRECOPY_FAILED, NULL);
         goto out;
     }
 
@@ -4010,11 +4017,6 @@ void migration_connect(MigrationState *s, Error *error_in)
     } else {
         /* This is a fresh new migration */
         rate_limit = migrate_max_bandwidth();
-
-        /* Notify before starting migration thread */
-        if (migration_call_notifiers(s, MIG_EVENT_PRECOPY_SETUP, &local_err)) {
-            goto fail;
-        }
     }
 
     migration_rate_set(rate_limit);
-- 
1.8.3.1

[RFC V2 3/8] vhost: reset vhost devices for cpr

[Steve Sistare]

When preserving a vhost fd using CPR, call VHOST_RESET_OWNER prior to CPR
in old QEMU.  Otherwise, new QEMU will fail when it calls VHOST_SET_OWNER
during vhost_dev_init.

Signed-off-by: Mark Kanda <mark.kanda@oracle.com>
Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
---
 include/hw/virtio/vhost.h |  1 +
 hw/virtio/vhost.c         | 32 ++++++++++++++++++++++++++++++++
 2 files changed, 33 insertions(+)

diff --git a/include/hw/virtio/vhost.h b/include/hw/virtio/vhost.h
index 38800a7..88a4838 100644
--- a/include/hw/virtio/vhost.h
+++ b/include/hw/virtio/vhost.h
@@ -132,6 +132,7 @@ struct vhost_dev {
     QLIST_ENTRY(vhost_dev) logdev_entry;
     QLIST_HEAD(, vhost_iommu) iommu_list;
     IOMMUNotifier n;
+    NotifierWithReturn cpr_transfer_notifier;
     const VhostDevConfigOps *config_ops;
 };
 
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index fc43853..a562e0c 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -24,6 +24,7 @@
 #include "standard-headers/linux/vhost_types.h"
 #include "hw/virtio/virtio-bus.h"
 #include "hw/mem/memory-device.h"
+#include "migration/misc.h"
 #include "migration/blocker.h"
 #include "migration/qemu-file-types.h"
 #include "system/dma.h"
@@ -1506,6 +1507,32 @@ static void vhost_virtqueue_cleanup(struct vhost_virtqueue *vq)
     }
 }
 
+static int vhost_cpr_notifier(NotifierWithReturn *notifier,
+                              MigrationEvent *e, Error **errp)
+{
+    struct vhost_dev *dev;
+    int r;
+
+    dev = container_of(notifier, struct vhost_dev, cpr_transfer_notifier);
+
+    if (dev->vhost_ops->backend_type != VHOST_BACKEND_TYPE_KERNEL) {
+        return 0;
+    }
+
+    if (e->type == MIG_EVENT_PRECOPY_SETUP) {
+        r = dev->vhost_ops->vhost_reset_device(dev);
+        if (r < 0) {
+            VHOST_OPS_DEBUG(r, "vhost_reset_device failed");
+        }
+    } else if (e->type == MIG_EVENT_PRECOPY_FAILED) {
+        r = dev->vhost_ops->vhost_set_owner(dev);
+        if (r < 0) {
+            VHOST_OPS_DEBUG(r, "vhost_set_owner failed");
+        }
+    }
+    return 0;
+}
+
 int vhost_dev_init(struct vhost_dev *hdev, void *opaque,
                    VhostBackendType backend_type, uint32_t busyloop_timeout,
                    Error **errp)
@@ -1516,6 +1543,7 @@ int vhost_dev_init(struct vhost_dev *hdev, void *opaque,
 
     hdev->vdev = NULL;
     hdev->migration_blocker = NULL;
+    hdev->cpr_transfer_notifier.notify = NULL;
 
     r = vhost_set_backend_type(hdev, backend_type);
     assert(r >= 0);
@@ -1616,6 +1644,9 @@ int vhost_dev_init(struct vhost_dev *hdev, void *opaque,
     hdev->log_enabled = false;
     hdev->started = false;
     memory_listener_register(&hdev->memory_listener, &address_space_memory);
+    migration_add_notifier_mode(&hdev->cpr_transfer_notifier,
+                                vhost_cpr_notifier,
+                                MIG_MODE_CPR_TRANSFER);
     QLIST_INSERT_HEAD(&vhost_devices, hdev, entry);
 
     /*
@@ -1672,6 +1703,7 @@ void vhost_dev_cleanup(struct vhost_dev *hdev)
         QLIST_REMOVE(hdev, entry);
     }
     migrate_del_blocker(&hdev->migration_blocker);
+    migration_remove_notifier(&hdev->cpr_transfer_notifier);
     g_free(hdev->mem);
     g_free(hdev->mem_sections);
     if (hdev->vhost_ops) {
-- 
1.8.3.1

Re: [RFC V2 3/8] vhost: reset vhost devices for cpr

[Vladimir Sementsov-Ogievskiy]

On 17.07.25 21:39, Steve Sistare wrote:
> When preserving a vhost fd using CPR, call VHOST_RESET_OWNER prior to CPR
> in old QEMU.  Otherwise, new QEMU will fail when it calls VHOST_SET_OWNER
> during vhost_dev_init.
> 

But, is there a benefit in passing vhostnet fd to target QEMU? As I understand,
we anyway need a full renitialization of vhost device, as it's just reset
on RESET_OWNER. So, is passing this fd better then reopen /dev/vhost-net
on target?


-- 
Best regards,
Vladimir

Re: [RFC V2 3/8] vhost: reset vhost devices for cpr

[Steven Sistare]

On 8/27/2025 7:29 AM, Vladimir Sementsov-Ogievskiy wrote:
> On 17.07.25 21:39, Steve Sistare wrote:
>> When preserving a vhost fd using CPR, call VHOST_RESET_OWNER prior to CPR
>> in old QEMU.  Otherwise, new QEMU will fail when it calls VHOST_SET_OWNER
>> during vhost_dev_init.
>>
> 
> But, is there a benefit in passing vhostnet fd to target QEMU? As I understand,
> we anyway need a full renitialization of vhost device, as it's just reset
> on RESET_OWNER. So, is passing this fd better then reopen /dev/vhost-net
> on target?

Some orchestraters pass pre-opened tap and vhost fd's on the qemu command line
when initially starting qemu.  For that model, preserving the open fd's in new
QEMU is the most natural solution.

- Steve

[RFC V2 4/8] cpr: delete all fds

[Steve Sistare]

Add the cpr_delete_fd_all function to delete all fds associated with a
device.

Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
---
 include/migration/cpr.h |  1 +
 migration/cpr.c         | 13 +++++++++++++
 2 files changed, 14 insertions(+)

diff --git a/include/migration/cpr.h b/include/migration/cpr.h
index 3fc19a7..0fa57dd 100644
--- a/include/migration/cpr.h
+++ b/include/migration/cpr.h
@@ -29,6 +29,7 @@ extern CprState cpr_state;
 
 void cpr_save_fd(const char *name, int id, int fd);
 void cpr_delete_fd(const char *name, int id);
+void cpr_delete_fd_all(const char *name);
 int cpr_find_fd(const char *name, int id);
 void cpr_resave_fd(const char *name, int id, int fd);
 int cpr_open_fd(const char *path, int flags, const char *name, int id,
diff --git a/migration/cpr.c b/migration/cpr.c
index 42ad0b0..e97be9d 100644
--- a/migration/cpr.c
+++ b/migration/cpr.c
@@ -83,6 +83,19 @@ void cpr_delete_fd(const char *name, int id)
     trace_cpr_delete_fd(name, id);
 }
 
+void cpr_delete_fd_all(const char *name)
+{
+    CprFd *elem, *next_elem;
+
+    QLIST_FOREACH_SAFE(elem, &cpr_state.fds, next, next_elem) {
+        if (!strcmp(elem->name, name)) {
+            QLIST_REMOVE(elem, next);
+            g_free(elem->name);
+            g_free(elem);
+        }
+    }
+}
+
 int cpr_find_fd(const char *name, int id)
 {
     CprFd *elem = find_fd(&cpr_state.fds, name, id);
-- 
1.8.3.1

[RFC V2 5/8] Revert "vhost-backend: remove vhost_kernel_reset_device()"

[Steve Sistare]

This reverts commit e6383293eb01928692047e617665a742cca87e23.
The reset function is needed for CPR.

Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
---
 hw/virtio/vhost-backend.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/hw/virtio/vhost-backend.c b/hw/virtio/vhost-backend.c
index 833804d..9b75141 100644
--- a/hw/virtio/vhost-backend.c
+++ b/hw/virtio/vhost-backend.c
@@ -221,6 +221,11 @@ static int vhost_kernel_set_owner(struct vhost_dev *dev)
     return vhost_kernel_call(dev, VHOST_SET_OWNER, NULL);
 }
 
+static int vhost_kernel_reset_device(struct vhost_dev *dev)
+{
+    return vhost_kernel_call(dev, VHOST_RESET_OWNER, NULL);
+}
+
 static int vhost_kernel_get_vq_index(struct vhost_dev *dev, int idx)
 {
     assert(idx >= dev->vq_index && idx < dev->vq_index + dev->nvqs);
@@ -345,6 +350,7 @@ const VhostOps kernel_ops = {
         .vhost_get_features = vhost_kernel_get_features,
         .vhost_set_backend_cap = vhost_kernel_set_backend_cap,
         .vhost_set_owner = vhost_kernel_set_owner,
+        .vhost_reset_device = vhost_kernel_reset_device,
         .vhost_get_vq_index = vhost_kernel_get_vq_index,
         .vhost_vsock_set_guest_cid = vhost_kernel_vsock_set_guest_cid,
         .vhost_vsock_set_running = vhost_kernel_vsock_set_running,
-- 
1.8.3.1

Re: [RFC V2 5/8] Revert "vhost-backend: remove vhost_kernel_reset_device()"

[Steven Sistare]

On 7/17/2025 2:39 PM, Steve Sistare wrote:
> This reverts commit e6383293eb01928692047e617665a742cca87e23.
> The reset function is needed for CPR.
> 
> Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
> ---
>   hw/virtio/vhost-backend.c | 6 ++++++
>   1 file changed, 6 insertions(+)

This is broken.
Drop this patch 5/8 and replace it with the following.
I still need to refactor this slightly versus patch
   [RFC V2 3/8] vhost: reset vhost devices for cpr
before posting V3.

- Steve

-------------------------
From: Steve Sistare <steven.sistare@oracle.com>
Date: Mon, 19 Aug 2024 11:16:27 -0700
Subject: [PATCH] vhost: vhost_reset_owner

Define vhost_reset_owner, needed for CPR.

Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
---
  hw/virtio/vhost-backend.c         | 6 ++++++
  hw/virtio/vhost.c                 | 4 ++--
  include/hw/virtio/vhost-backend.h | 1 +
  3 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/hw/virtio/vhost-backend.c b/hw/virtio/vhost-backend.c
index 833804d..54fc8ee 100644
--- a/hw/virtio/vhost-backend.c
+++ b/hw/virtio/vhost-backend.c
@@ -221,6 +221,11 @@ static int vhost_kernel_set_owner(struct vhost_dev *dev)
      return vhost_kernel_call(dev, VHOST_SET_OWNER, NULL);
  }

+static int vhost_kernel_reset_owner(struct vhost_dev *dev)
+{
+    return vhost_kernel_call(dev, VHOST_RESET_OWNER, NULL);
+}
+
  static int vhost_kernel_get_vq_index(struct vhost_dev *dev, int idx)
  {
      assert(idx >= dev->vq_index && idx < dev->vq_index + dev->nvqs);
@@ -345,6 +350,7 @@ const VhostOps kernel_ops = {
          .vhost_get_features = vhost_kernel_get_features,
          .vhost_set_backend_cap = vhost_kernel_set_backend_cap,
          .vhost_set_owner = vhost_kernel_set_owner,
+        .vhost_reset_owner = vhost_kernel_reset_owner,
          .vhost_get_vq_index = vhost_kernel_get_vq_index,
          .vhost_vsock_set_guest_cid = vhost_kernel_vsock_set_guest_cid,
          .vhost_vsock_set_running = vhost_kernel_vsock_set_running,
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index 40d5cd8..72e010f 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -1522,9 +1522,9 @@ static int vhost_cpr_notifier(NotifierWithReturn *notifier,
      }

      if (e->type == MIG_EVENT_PRECOPY_SETUP) {
-        r = dev->vhost_ops->vhost_reset_device(dev);
+        r = dev->vhost_ops->vhost_reset_owner(dev);
          if (r < 0) {
-            VHOST_OPS_DEBUG(r, "vhost_reset_device failed");
+            VHOST_OPS_DEBUG(r, "vhost_reset_owner failed");
          }
      } else if (e->type == MIG_EVENT_PRECOPY_FAILED) {
          r = dev->vhost_ops->vhost_set_owner(dev);
diff --git a/include/hw/virtio/vhost-backend.h b/include/hw/virtio/vhost-backend.h
index d6df209..5b68bfb 100644
--- a/include/hw/virtio/vhost-backend.h
+++ b/include/hw/virtio/vhost-backend.h
@@ -190,6 +190,7 @@ typedef struct VhostOps {
      vhost_get_features_op vhost_get_features;
      vhost_set_backend_cap_op vhost_set_backend_cap;
      vhost_set_owner_op vhost_set_owner;
+    vhost_set_owner_op vhost_reset_owner;
      vhost_reset_device_op vhost_reset_device;
      vhost_get_vq_index_op vhost_get_vq_index;
      vhost_set_vring_enable_op vhost_set_vring_enable;
--
1.8.3.1

-------------------------

[RFC V2 6/8] tap: common return label

[Steve Sistare]

Modify net_init_tap so every return branches to a common label, for
common cleanup in a subsequent patch.  No functional change.

Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
---
 net/tap.c | 54 ++++++++++++++++++++++++++++++++++++------------------
 1 file changed, 36 insertions(+), 18 deletions(-)

diff --git a/net/tap.c b/net/tap.c
index 35552c4..1b239fd 100644
--- a/net/tap.c
+++ b/net/tap.c
@@ -780,7 +780,8 @@ int net_init_tap(const Netdev *netdev, const char *name,
      * For -netdev, peer is always NULL. */
     if (peer && (tap->has_queues || tap->fds || tap->vhostfds)) {
         error_setg(errp, "Multiqueue tap cannot be used with hubs");
-        return -1;
+        ret = -1;
+        goto out;
     }
 
     if (tap->fd) {
@@ -790,25 +791,29 @@ int net_init_tap(const Netdev *netdev, const char *name,
             error_setg(errp, "ifname=, script=, downscript=, vnet_hdr=, "
                        "helper=, queues=, fds=, and vhostfds= "
                        "are invalid with fd=");
-            return -1;
+            ret = -1;
+            goto out;
         }
 
         fd = monitor_fd_param(monitor_cur(), tap->fd, errp);
         if (fd == -1) {
-            return -1;
+            ret = -1;
+            goto out;
         }
 
         if (!g_unix_set_fd_nonblocking(fd, true, NULL)) {
             error_setg_errno(errp, errno, "%s: Can't use file descriptor %d",
                              name, fd);
             close(fd);
-            return -1;
+            ret = -1;
+            goto out;
         }
 
         vnet_hdr = tap_probe_vnet_hdr(fd, errp);
         if (vnet_hdr < 0) {
             close(fd);
-            return -1;
+            ret = -1;
+            goto out;
         }
 
         net_init_tap_one(tap, peer, "tap", name, NULL,
@@ -817,7 +822,8 @@ int net_init_tap(const Netdev *netdev, const char *name,
         if (err) {
             error_propagate(errp, err);
             close(fd);
-            return -1;
+            ret = -1;
+            goto out;
         }
     } else if (tap->fds) {
         char **fds;
@@ -830,7 +836,8 @@ int net_init_tap(const Netdev *netdev, const char *name,
             error_setg(errp, "ifname=, script=, downscript=, vnet_hdr=, "
                        "helper=, queues=, and vhostfd= "
                        "are invalid with fds=");
-            return -1;
+            ret = -1;
+            goto out;
         }
 
         fds = g_new0(char *, MAX_TAP_QUEUES);
@@ -894,30 +901,35 @@ free_fail:
         }
         g_free(fds);
         g_free(vhost_fds);
-        return ret;
+        goto out;
+
     } else if (tap->helper) {
         if (tap->ifname || tap->script || tap->downscript ||
             tap->has_vnet_hdr || tap->has_queues || tap->vhostfds) {
             error_setg(errp, "ifname=, script=, downscript=, vnet_hdr=, "
                        "queues=, and vhostfds= are invalid with helper=");
-            return -1;
+            ret = -1;
+            goto out;
         }
 
         fd = net_bridge_run_helper(tap->helper,
                                    tap->br ?: DEFAULT_BRIDGE_INTERFACE,
                                    errp);
         if (fd == -1) {
-            return -1;
+            ret = -1;
+            goto out;
         }
 
         if (!g_unix_set_fd_nonblocking(fd, true, NULL)) {
             error_setg_errno(errp, errno, "Failed to set FD nonblocking");
-            return -1;
+            ret = -1;
+            goto out;
         }
         vnet_hdr = tap_probe_vnet_hdr(fd, errp);
         if (vnet_hdr < 0) {
             close(fd);
-            return -1;
+            ret = -1;
+            goto out;
         }
 
         net_init_tap_one(tap, peer, "bridge", name, ifname,
@@ -926,14 +938,16 @@ free_fail:
         if (err) {
             error_propagate(errp, err);
             close(fd);
-            return -1;
+            ret = -1;
+            goto out;
         }
     } else {
         g_autofree char *default_script = NULL;
         g_autofree char *default_downscript = NULL;
         if (tap->vhostfds) {
             error_setg(errp, "vhostfds= is invalid if fds= wasn't specified");
-            return -1;
+            ret = -1;
+            goto out;
         }
 
         if (!script) {
@@ -954,14 +968,16 @@ free_fail:
             fd = net_tap_init(tap, &vnet_hdr, i >= 1 ? "no" : script,
                               ifname, sizeof ifname, queues > 1, errp);
             if (fd == -1) {
-                return -1;
+                ret = -1;
+                goto out;
             }
 
             if (queues > 1 && i == 0 && !tap->ifname) {
                 if (tap_fd_get_ifname(fd, ifname)) {
                     error_setg(errp, "Fail to get ifname");
                     close(fd);
-                    return -1;
+                    ret = -1;
+                    goto out;
                 }
             }
 
@@ -972,12 +988,14 @@ free_fail:
             if (err) {
                 error_propagate(errp, err);
                 close(fd);
-                return -1;
+                ret = -1;
+                goto out;
             }
         }
     }
 
-    return 0;
+out:
+    return ret;
 }
 
 VHostNetState *tap_get_vhost_net(NetClientState *nc)
-- 
1.8.3.1

[RFC V2 7/8] tap: cpr support

[Steve Sistare]

Provide the cpr=on option to preserve TAP and vhost descriptors during
cpr-transfer, so the management layer does not need to create a new
device for the target.

Save all tap fd's in canonical order, leveraging the index argument of
cpr_save_fd.  For the i'th queue, the tap device fd is saved at index 2*i,
and the vhostfd (if any) at index 2*i+1.

tap and vhost fd's are passed by name to the monitor when a NIC is hot
plugged, but the name is not known to qemu after cpr.  Allow the manager
to pass -1 for the fd "name" in the new qemu args to indicate that QEMU
should search for a saved value.  Example:

  -netdev tap,id=hostnet2,fds=-1:-1,vhostfds=-1:-1,cpr=on

Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
---
 qapi/net.json           |  5 +++-
 include/migration/cpr.h |  2 +-
 hw/vfio/device.c        |  2 +-
 migration/cpr.c         | 11 ++++----
 net/tap.c               | 70 ++++++++++++++++++++++++++++++++++++++-----------
 5 files changed, 67 insertions(+), 23 deletions(-)

diff --git a/qapi/net.json b/qapi/net.json
index 97ea183..5c7422b 100644
--- a/qapi/net.json
+++ b/qapi/net.json
@@ -238,6 +238,8 @@
 # @poll-us: maximum number of microseconds that could be spent on busy
 #     polling for tap (since 2.7)
 #
+# @cpr: preserve fds and vhostfds during cpr-transfer.
+#
 # Since: 1.2
 ##
 { 'struct': 'NetdevTapOptions',
@@ -256,7 +258,8 @@
     '*vhostfds':   'str',
     '*vhostforce': 'bool',
     '*queues':     'uint32',
-    '*poll-us':    'uint32'} }
+    '*poll-us':    'uint32',
+    '*cpr':        'bool'} }
 
 ##
 # @NetdevSocketOptions:
diff --git a/include/migration/cpr.h b/include/migration/cpr.h
index 0fa57dd..baff57f 100644
--- a/include/migration/cpr.h
+++ b/include/migration/cpr.h
@@ -45,7 +45,7 @@ void cpr_state_close(void);
 struct QIOChannel *cpr_state_ioc(void);
 
 bool cpr_incoming_needed(void *opaque);
-int cpr_get_fd_param(const char *name, const char *fdname, int index,
+int cpr_get_fd_param(const char *name, const char *fdname, int index, bool cpr,
                      Error **errp);
 
 QEMUFile *cpr_transfer_output(MigrationChannel *channel, Error **errp);
diff --git a/hw/vfio/device.c b/hw/vfio/device.c
index 96cf214..9eb6699 100644
--- a/hw/vfio/device.c
+++ b/hw/vfio/device.c
@@ -351,7 +351,7 @@ void vfio_device_free_name(VFIODevice *vbasedev)
 
 void vfio_device_set_fd(VFIODevice *vbasedev, const char *str, Error **errp)
 {
-    vbasedev->fd = cpr_get_fd_param(vbasedev->dev->id, str, 0, errp);
+    vbasedev->fd = cpr_get_fd_param(vbasedev->dev->id, str, 0, true, errp);
 }
 
 static VFIODeviceIOOps vfio_device_io_ops_ioctl;
diff --git a/migration/cpr.c b/migration/cpr.c
index e97be9d..6d01b8c 100644
--- a/migration/cpr.c
+++ b/migration/cpr.c
@@ -282,6 +282,7 @@ bool cpr_incoming_needed(void *opaque)
  * @name: CPR name for the descriptor
  * @fdname: An integer-valued string, or a name passed to a getfd command
  * @index: CPR index of the descriptor
+ * @cpr: use cpr
  * @errp: returned error message
  *
  * If CPR is not being performed, then use @fdname to find the fd.
@@ -291,22 +292,22 @@ bool cpr_incoming_needed(void *opaque)
  * On success returns the fd value, else returns -1.
  */
 int cpr_get_fd_param(const char *name, const char *fdname, int index,
-                     Error **errp)
+                     bool cpr, Error **errp)
 {
     ERRP_GUARD();
     int fd;
 
-    if (cpr_is_incoming()) {
+    if (cpr && cpr_is_incoming()) {
         fd = cpr_find_fd(name, index);
         if (fd < 0) {
             error_setg(errp, "cannot find saved value for fd %s", fdname);
         }
     } else {
         fd = monitor_fd_param(monitor_cur(), fdname, errp);
-        if (fd >= 0) {
-            cpr_save_fd(name, index, fd);
-        } else {
+        if (fd < 0) {
             error_prepend(errp, "Could not parse object fd %s:", fdname);
+        } else if (cpr) {
+            cpr_save_fd(name, index, fd);
         }
     }
     return fd;
diff --git a/net/tap.c b/net/tap.c
index 1b239fd..6a12751 100644
--- a/net/tap.c
+++ b/net/tap.c
@@ -35,6 +35,7 @@
 #include "net/eth.h"
 #include "net/net.h"
 #include "clients.h"
+#include "migration/cpr.h"
 #include "monitor/monitor.h"
 #include "system/system.h"
 #include "qapi/error.h"
@@ -59,6 +60,7 @@ typedef struct TAPState {
     bool has_ufo;
     bool has_uso;
     bool enabled;
+    bool cpr;
     VHostNetState *vhost_net;
     unsigned host_vnet_hdr_len;
     Notifier exit;
@@ -290,6 +292,9 @@ static void tap_cleanup(NetClientState *nc)
 {
     TAPState *s = DO_UPCAST(TAPState, nc, nc);
 
+    if (s->cpr) {
+        cpr_delete_fd_all(nc->name);
+    }
     if (s->vhost_net) {
         vhost_net_cleanup(s->vhost_net);
         g_free(s->vhost_net);
@@ -642,18 +647,24 @@ static int net_tap_init(const NetdevTapOptions *tap, int *vnet_hdr,
     return fd;
 }
 
+/* CPR fd's for each queue are saved at these indices */
+#define TAP_FD_INDEX(queue)         (2 * (queue) + 0)
+#define TAP_VHOSTFD_INDEX(queue)    (2 * (queue) + 1)
+
 #define MAX_TAP_QUEUES 1024
 
 static void net_init_tap_one(const NetdevTapOptions *tap, NetClientState *peer,
                              const char *model, const char *name,
                              const char *ifname, const char *script,
                              const char *downscript, const char *vhostfdname,
-                             int vnet_hdr, int fd, Error **errp)
+                             int vnet_hdr, int fd, int index, Error **errp)
 {
     Error *err = NULL;
     TAPState *s = net_tap_fd_init(peer, model, name, fd, vnet_hdr);
+    bool cpr = tap->has_cpr ? tap->cpr : false;
     int vhostfd;
 
+    s->cpr = cpr;
     tap_set_sndbuf(s->fd, tap, &err);
     if (err) {
         error_propagate(errp, err);
@@ -688,7 +699,7 @@ static void net_init_tap_one(const NetdevTapOptions *tap, NetClientState *peer,
         }
 
         if (vhostfdname) {
-            vhostfd = monitor_fd_param(monitor_cur(), vhostfdname, &err);
+            vhostfd = cpr_get_fd_param(name, vhostfdname, index, cpr, &err);
             if (vhostfd == -1) {
                 error_propagate(errp, err);
                 goto failed;
@@ -699,7 +710,13 @@ static void net_init_tap_one(const NetdevTapOptions *tap, NetClientState *peer,
                 goto failed;
             }
         } else {
-            vhostfd = open("/dev/vhost-net", O_RDWR);
+            vhostfd = cpr ? cpr_find_fd(name, index) : -1;
+            if (vhostfd < 0) {
+                vhostfd = open("/dev/vhost-net", O_RDWR);
+                if (cpr && vhostfd >= 0) {
+                    cpr_save_fd(name, index, vhostfd);
+                }
+            }
             if (vhostfd < 0) {
                 error_setg_errno(errp, errno,
                                  "tap: open vhost char device failed");
@@ -727,6 +744,9 @@ static void net_init_tap_one(const NetdevTapOptions *tap, NetClientState *peer,
     return;
 
 failed:
+    if (cpr) {
+        cpr_delete_fd_all(name);
+    }
     qemu_del_net_client(&s->nc);
 }
 
@@ -759,7 +779,8 @@ static int get_fds(char *str, char *fds[], int max)
 int net_init_tap(const Netdev *netdev, const char *name,
                  NetClientState *peer, Error **errp)
 {
-    const NetdevTapOptions *tap;
+    const NetdevTapOptions *tap = &netdev->u.tap;
+    bool cpr = tap->has_cpr ? tap->cpr : false;
     int fd, vnet_hdr = 0, i = 0, queues;
     /* for the no-fd, no-helper case */
     const char *script;
@@ -795,7 +816,7 @@ int net_init_tap(const Netdev *netdev, const char *name,
             goto out;
         }
 
-        fd = monitor_fd_param(monitor_cur(), tap->fd, errp);
+        fd = cpr_get_fd_param(name, tap->fd, TAP_FD_INDEX(0), cpr, errp);
         if (fd == -1) {
             ret = -1;
             goto out;
@@ -818,13 +839,14 @@ int net_init_tap(const Netdev *netdev, const char *name,
 
         net_init_tap_one(tap, peer, "tap", name, NULL,
                          script, downscript,
-                         vhostfdname, vnet_hdr, fd, &err);
+                         vhostfdname, vnet_hdr, fd, TAP_VHOSTFD_INDEX(0), &err);
         if (err) {
             error_propagate(errp, err);
             close(fd);
             ret = -1;
             goto out;
         }
+
     } else if (tap->fds) {
         char **fds;
         char **vhost_fds;
@@ -855,7 +877,7 @@ int net_init_tap(const Netdev *netdev, const char *name,
         }
 
         for (i = 0; i < nfds; i++) {
-            fd = monitor_fd_param(monitor_cur(), fds[i], errp);
+            fd = cpr_get_fd_param(name, fds[i], TAP_FD_INDEX(i), cpr, errp);
             if (fd == -1) {
                 ret = -1;
                 goto free_fail;
@@ -884,7 +906,7 @@ int net_init_tap(const Netdev *netdev, const char *name,
             net_init_tap_one(tap, peer, "tap", name, ifname,
                              script, downscript,
                              tap->vhostfds ? vhost_fds[i] : NULL,
-                             vnet_hdr, fd, &err);
+                             vnet_hdr, fd, TAP_VHOSTFD_INDEX(i), &err);
             if (err) {
                 error_propagate(errp, err);
                 ret = -1;
@@ -912,9 +934,15 @@ free_fail:
             goto out;
         }
 
-        fd = net_bridge_run_helper(tap->helper,
-                                   tap->br ?: DEFAULT_BRIDGE_INTERFACE,
-                                   errp);
+        fd = cpr ? cpr_find_fd(name, TAP_FD_INDEX(0)) : -1;
+        if (fd < 0) {
+            fd = net_bridge_run_helper(tap->helper,
+                                    tap->br ?: DEFAULT_BRIDGE_INTERFACE,
+                                    errp);
+            if (cpr && fd >= 0) {
+                cpr_save_fd(name, TAP_FD_INDEX(0), fd);
+            }
+        }
         if (fd == -1) {
             ret = -1;
             goto out;
@@ -934,13 +962,14 @@ free_fail:
 
         net_init_tap_one(tap, peer, "bridge", name, ifname,
                          script, downscript, vhostfdname,
-                         vnet_hdr, fd, &err);
+                         vnet_hdr, fd, TAP_VHOSTFD_INDEX(0), &err);
         if (err) {
             error_propagate(errp, err);
             close(fd);
             ret = -1;
             goto out;
         }
+
     } else {
         g_autofree char *default_script = NULL;
         g_autofree char *default_downscript = NULL;
@@ -965,8 +994,14 @@ free_fail:
         }
 
         for (i = 0; i < queues; i++) {
-            fd = net_tap_init(tap, &vnet_hdr, i >= 1 ? "no" : script,
-                              ifname, sizeof ifname, queues > 1, errp);
+            fd = cpr ? cpr_find_fd(name, TAP_FD_INDEX(i)) : -1;
+            if (fd < 0) {
+                fd = net_tap_init(tap, &vnet_hdr, i >= 1 ? "no" : script,
+                                ifname, sizeof ifname, queues > 1, errp);
+                if (cpr && fd >= 0) {
+                    cpr_save_fd(name, TAP_FD_INDEX(i), fd);
+                }
+            }
             if (fd == -1) {
                 ret = -1;
                 goto out;
@@ -984,7 +1019,9 @@ free_fail:
             net_init_tap_one(tap, peer, "tap", name, ifname,
                              i >= 1 ? "no" : script,
                              i >= 1 ? "no" : downscript,
-                             vhostfdname, vnet_hdr, fd, &err);
+                             vhostfdname, vnet_hdr,
+                             fd, TAP_VHOSTFD_INDEX(i),
+                             &err);
             if (err) {
                 error_propagate(errp, err);
                 close(fd);
@@ -995,6 +1032,9 @@ free_fail:
     }
 
 out:
+    if (ret && cpr) {
+        cpr_delete_fd_all(name);
+    }
     return ret;
 }
 
-- 
1.8.3.1

[RFC V2 8/8] tap: postload fix for cpr

[Steve Sistare]

After cpr of a multi-queue NIC, if any queues are unused, then the
corresponding tap is marked enabled in userland, but it is disabled in the
kernel for the fd that was preserved.  One cannot call tap_disable() during
postload, because that eventually calls IFF_DETACH_QUEUE, which fails
because the queue is already detached.  Define tap_disable_postload to
avoid IFF_DETACH_QUEUE.

Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
---
 include/net/tap.h   |  1 +
 hw/net/virtio-net.c | 20 ++++++++++++++++++++
 net/tap-win32.c     |  5 +++++
 net/tap.c           | 17 +++++++++++++++++
 4 files changed, 43 insertions(+)

diff --git a/include/net/tap.h b/include/net/tap.h
index 5d58551..9456abe 100644
--- a/include/net/tap.h
+++ b/include/net/tap.h
@@ -30,6 +30,7 @@
 
 int tap_enable(NetClientState *nc);
 int tap_disable(NetClientState *nc);
+void tap_disable_postload(NetClientState *nc);
 
 int tap_get_fd(NetClientState *nc);
 
diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index eb93607..b45128e 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -730,6 +730,25 @@ static int peer_detach(VirtIONet *n, int index)
     return tap_disable(nc->peer);
 }
 
+/*
+ * Set the disabled flag on unused queue pairs after vmstate load, without
+ * calling IFF_DETACH_QUEUE, which fails because the queue is already detached.
+ */
+static void virtio_net_postload_queue_pairs(VirtIONet *n)
+{
+    int i;
+    MigMode mode = migrate_mode();
+
+    if (mode == MIG_MODE_CPR_TRANSFER) {
+        for (i = n->curr_queue_pairs; i < n->max_queue_pairs; i++) {
+            NetClientState *nc = qemu_get_subqueue(n->nic, i);
+            if (nc->peer && nc->peer->info->type == NET_CLIENT_DRIVER_TAP) {
+                tap_disable_postload(nc->peer);
+            }
+        }
+    }
+}
+
 static void virtio_net_set_queue_pairs(VirtIONet *n)
 {
     int i;
@@ -3106,6 +3125,7 @@ static int virtio_net_post_load_device(void *opaque, int version_id)
      */
     n->saved_guest_offloads = n->curr_guest_offloads;
 
+    virtio_net_postload_queue_pairs(n);
     virtio_net_set_queue_pairs(n);
 
     /* Find the first multicast entry in the saved MAC filter */
diff --git a/net/tap-win32.c b/net/tap-win32.c
index 671dee9..66be7c9 100644
--- a/net/tap-win32.c
+++ b/net/tap-win32.c
@@ -771,3 +771,8 @@ int tap_disable(NetClientState *nc)
 {
     abort();
 }
+
+void tap_disable_postload(NetClientState *nc)
+{
+    abort();
+}
diff --git a/net/tap.c b/net/tap.c
index 6a12751..c7f9023 100644
--- a/net/tap.c
+++ b/net/tap.c
@@ -1079,3 +1079,20 @@ int tap_disable(NetClientState *nc)
         return ret;
     }
 }
+
+/*
+ * On cpr restart, the tap is marked enabled in userland, but it might be
+ * disabled in the kernel, and IFF_DETACH_QUEUE will fail because it is
+ * already detached.  This function disables without calling IFF_DETACH_QUEUE.
+ */
+void tap_disable_postload(NetClientState *nc)
+{
+    TAPState *s = DO_UPCAST(TAPState, nc, nc);
+
+    if (!s->cpr || s->enabled == 0) {
+        return;
+    } else {
+        s->enabled = false;
+        tap_update_fd_handler(s);
+    }
+}
-- 
1.8.3.1

Re: [RFC V2 0/8] Live update: tap and vhost

[Lei Yang]

Hi Steve

I tested your patch which hit a problem under enable/disable nic mq
state(The full test scenario is live migration vm at local under
enable/disable vm nic mq state):
Run command: /qemu-img info /home/images/vm1.qcow2 --output=json
Error info: qemu-img: Could not open '/home/images/vm1.qcow2': Failed
to get shared "write" lock

In the same environment if I remove the enable/disable nic mq steps,
it can work well.

Note: This issue has not been reproduced 100%, but in order to confirm
this is really a bug, I test the following:
1. Live migration vm at local under enable/disable nic mq state: 3/5
reproduce ratio
2. Only live migration 5 times are all passed.

Thanks
Lei

On Fri, Jul 18, 2025 at 4:48 AM Steve Sistare <steven.sistare@oracle.com> wrote:
>
> Tap and vhost devices can be preserved during cpr-transfer using
> traditional live migration methods, wherein the management layer
> creates new interfaces for the target and fiddles with 'ip link'
> to deactivate the old interface and activate the new.
>
> However, CPR can simply send the file descriptors to new QEMU,
> with no special management actions required.  The user enables
> this behavior by specifing '-netdev tap,cpr=on'.  The default
> is cpr=off.
>
> Steve Sistare (8):
>   migration: stop vm earlier for cpr
>   migration: cpr setup notifier
>   vhost: reset vhost devices for cpr
>   cpr: delete all fds
>   Revert "vhost-backend: remove vhost_kernel_reset_device()"
>   tap: common return label
>   tap: cpr support
>   tap: postload fix for cpr
>
>  qapi/net.json             |   5 +-
>  include/hw/virtio/vhost.h |   1 +
>  include/migration/cpr.h   |   3 +-
>  include/net/tap.h         |   1 +
>  hw/net/virtio-net.c       |  20 +++++++
>  hw/vfio/device.c          |   2 +-
>  hw/virtio/vhost-backend.c |   6 ++
>  hw/virtio/vhost.c         |  32 +++++++++++
>  migration/cpr.c           |  24 ++++++--
>  migration/migration.c     |  38 ++++++++-----
>  net/tap-win32.c           |   5 ++
>  net/tap.c                 | 141 +++++++++++++++++++++++++++++++++++-----------
>  12 files changed, 223 insertions(+), 55 deletions(-)
>
> --
> 1.8.3.1
>
>

Re: [RFC V2 0/8] Live update: tap and vhost

[Steven Sistare]

On 7/18/2025 4:48 AM, Lei Yang wrote:
> Hi Steve
> 
> I tested your patch which hit a problem under enable/disable nic mq
> state(The full test scenario is live migration vm at local under
> enable/disable vm nic mq state):
> Run command: /qemu-img info /home/images/vm1.qcow2 --output=json
> Error info: qemu-img: Could not open '/home/images/vm1.qcow2': Failed
> to get shared "write" lock
> 
> In the same environment if I remove the enable/disable nic mq steps,
> it can work well.
> 
> Note: This issue has not been reproduced 100%, but in order to confirm
> this is really a bug, I test the following:
> 1. Live migration vm at local under enable/disable nic mq state: 3/5
> reproduce ratio
> 2. Only live migration 5 times are all passed.

Thank you Lei for testing this.  I would like to debug it, but I need
more information:

* did you add the cpr=on option for tap devices, or is this just
   a regression test of existing functionality?

* What platform: linux or windows?

* Is this a publicly available test that I can run?  I cannot find any
   reference to "Live migration vm at local under enable/disable nic mq state"

If this is not a public test, can you provide more detail on the commands
issued at each step in the test?

Thanks!

- Steve

> On Fri, Jul 18, 2025 at 4:48 AM Steve Sistare <steven.sistare@oracle.com> wrote:
>>
>> Tap and vhost devices can be preserved during cpr-transfer using
>> traditional live migration methods, wherein the management layer
>> creates new interfaces for the target and fiddles with 'ip link'
>> to deactivate the old interface and activate the new.
>>
>> However, CPR can simply send the file descriptors to new QEMU,
>> with no special management actions required.  The user enables
>> this behavior by specifing '-netdev tap,cpr=on'.  The default
>> is cpr=off.
>>
>> Steve Sistare (8):
>>    migration: stop vm earlier for cpr
>>    migration: cpr setup notifier
>>    vhost: reset vhost devices for cpr
>>    cpr: delete all fds
>>    Revert "vhost-backend: remove vhost_kernel_reset_device()"
>>    tap: common return label
>>    tap: cpr support
>>    tap: postload fix for cpr
>>
>>   qapi/net.json             |   5 +-
>>   include/hw/virtio/vhost.h |   1 +
>>   include/migration/cpr.h   |   3 +-
>>   include/net/tap.h         |   1 +
>>   hw/net/virtio-net.c       |  20 +++++++
>>   hw/vfio/device.c          |   2 +-
>>   hw/virtio/vhost-backend.c |   6 ++
>>   hw/virtio/vhost.c         |  32 +++++++++++
>>   migration/cpr.c           |  24 ++++++--
>>   migration/migration.c     |  38 ++++++++-----
>>   net/tap-win32.c           |   5 ++
>>   net/tap.c                 | 141 +++++++++++++++++++++++++++++++++++-----------
>>   12 files changed, 223 insertions(+), 55 deletions(-)
>>
>> --
>> 1.8.3.1
>>
>>
> 

Re: [RFC V2 0/8] Live update: tap and vhost

[Lei Yang]

On Fri, Jul 18, 2025 at 4:48 PM Lei Yang <leiyang@redhat.com> wrote:
>
> Hi Steve
>
> I tested your patch which hit a problem under enable/disable nic mq
> state(The full test scenario is live migration vm at local under
> enable/disable vm nic mq state):
> Run command: /qemu-img info /home/images/vm1.qcow2 --output=json
> Error info: qemu-img: Could not open '/home/images/vm1.qcow2': Failed
> to get shared "write" lock
>
> In the same environment if I remove the enable/disable nic mq steps,
> it can work well.
>
> Note: This issue has not been reproduced 100%, but in order to confirm
> this is really a bug, I test the following:
> 1. Live migration vm at local under enable/disable nic mq state: 3/5
> reproduce ratio
> 2. Only live migration 5 times are all passed.
>
After double confirm tests, the above mentioned problem should not be
related to the current series of patches, because I also hit this
problem after multiple tests using the upstream master branch.

Tested-by: Lei Yang <leiyang@redhat.com>
> Thanks
> Lei
>
> On Fri, Jul 18, 2025 at 4:48 AM Steve Sistare <steven.sistare@oracle.com> wrote:
> >
> > Tap and vhost devices can be preserved during cpr-transfer using
> > traditional live migration methods, wherein the management layer
> > creates new interfaces for the target and fiddles with 'ip link'
> > to deactivate the old interface and activate the new.
> >
> > However, CPR can simply send the file descriptors to new QEMU,
> > with no special management actions required.  The user enables
> > this behavior by specifing '-netdev tap,cpr=on'.  The default
> > is cpr=off.
> >
> > Steve Sistare (8):
> >   migration: stop vm earlier for cpr
> >   migration: cpr setup notifier
> >   vhost: reset vhost devices for cpr
> >   cpr: delete all fds
> >   Revert "vhost-backend: remove vhost_kernel_reset_device()"
> >   tap: common return label
> >   tap: cpr support
> >   tap: postload fix for cpr
> >
> >  qapi/net.json             |   5 +-
> >  include/hw/virtio/vhost.h |   1 +
> >  include/migration/cpr.h   |   3 +-
> >  include/net/tap.h         |   1 +
> >  hw/net/virtio-net.c       |  20 +++++++
> >  hw/vfio/device.c          |   2 +-
> >  hw/virtio/vhost-backend.c |   6 ++
> >  hw/virtio/vhost.c         |  32 +++++++++++
> >  migration/cpr.c           |  24 ++++++--
> >  migration/migration.c     |  38 ++++++++-----
> >  net/tap-win32.c           |   5 ++
> >  net/tap.c                 | 141 +++++++++++++++++++++++++++++++++++-----------
> >  12 files changed, 223 insertions(+), 55 deletions(-)
> >
> > --
> > 1.8.3.1
> >
> >

Re: [RFC V2 0/8] Live update: tap and vhost

[Fabiano Rosas]

Steve Sistare <steven.sistare@oracle.com> writes:

> Tap and vhost devices can be preserved during cpr-transfer using
> traditional live migration methods, wherein the management layer
> creates new interfaces for the target and fiddles with 'ip link'
> to deactivate the old interface and activate the new.
>
> However, CPR can simply send the file descriptors to new QEMU,
> with no special management actions required.  The user enables
> this behavior by specifing '-netdev tap,cpr=on'.  The default
> is cpr=off.
>
> Steve Sistare (8):
>   migration: stop vm earlier for cpr
>   migration: cpr setup notifier
>   vhost: reset vhost devices for cpr
>   cpr: delete all fds
>   Revert "vhost-backend: remove vhost_kernel_reset_device()"
>   tap: common return label
>   tap: cpr support
>   tap: postload fix for cpr
>
>  qapi/net.json             |   5 +-
>  include/hw/virtio/vhost.h |   1 +
>  include/migration/cpr.h   |   3 +-
>  include/net/tap.h         |   1 +
>  hw/net/virtio-net.c       |  20 +++++++
>  hw/vfio/device.c          |   2 +-
>  hw/virtio/vhost-backend.c |   6 ++
>  hw/virtio/vhost.c         |  32 +++++++++++
>  migration/cpr.c           |  24 ++++++--
>  migration/migration.c     |  38 ++++++++-----
>  net/tap-win32.c           |   5 ++
>  net/tap.c                 | 141 +++++++++++++++++++++++++++++++++++-----------
>  12 files changed, 223 insertions(+), 55 deletions(-)

Hi Steve,

Patches 1-2 seem to potentially interact with your arm pending
interrupts fix. Do we want them together?

Re: [RFC V2 0/8] Live update: tap and vhost

[Steven Sistare]

On 8/5/2025 9:54 AM, Fabiano Rosas wrote:
> Steve Sistare <steven.sistare@oracle.com> writes:
> 
>> Tap and vhost devices can be preserved during cpr-transfer using
>> traditional live migration methods, wherein the management layer
>> creates new interfaces for the target and fiddles with 'ip link'
>> to deactivate the old interface and activate the new.
>>
>> However, CPR can simply send the file descriptors to new QEMU,
>> with no special management actions required.  The user enables
>> this behavior by specifing '-netdev tap,cpr=on'.  The default
>> is cpr=off.
>>
>> Steve Sistare (8):
>>    migration: stop vm earlier for cpr
>>    migration: cpr setup notifier
>>    vhost: reset vhost devices for cpr
>>    cpr: delete all fds
>>    Revert "vhost-backend: remove vhost_kernel_reset_device()"
>>    tap: common return label
>>    tap: cpr support
>>    tap: postload fix for cpr
>>
>>   qapi/net.json             |   5 +-
>>   include/hw/virtio/vhost.h |   1 +
>>   include/migration/cpr.h   |   3 +-
>>   include/net/tap.h         |   1 +
>>   hw/net/virtio-net.c       |  20 +++++++
>>   hw/vfio/device.c          |   2 +-
>>   hw/virtio/vhost-backend.c |   6 ++
>>   hw/virtio/vhost.c         |  32 +++++++++++
>>   migration/cpr.c           |  24 ++++++--
>>   migration/migration.c     |  38 ++++++++-----
>>   net/tap-win32.c           |   5 ++
>>   net/tap.c                 | 141 +++++++++++++++++++++++++++++++++++-----------
>>   12 files changed, 223 insertions(+), 55 deletions(-)
> 
> Hi Steve,
> 
> Patches 1-2 seem to potentially interact with your arm pending
> interrupts fix. Do we want them together?

Good observation, thanks!.  I may need patches 1-2 to completely close
the dropped interrupt race.  I will do more testing to verify that.

- Steve

Re: [RFC V2 0/8] Live update: tap and vhost

[Peter Xu]

On Tue, Aug 05, 2025 at 03:53:09PM -0400, Steven Sistare wrote:
> On 8/5/2025 9:54 AM, Fabiano Rosas wrote:
> > Steve Sistare <steven.sistare@oracle.com> writes:
> > 
> > > Tap and vhost devices can be preserved during cpr-transfer using
> > > traditional live migration methods, wherein the management layer
> > > creates new interfaces for the target and fiddles with 'ip link'
> > > to deactivate the old interface and activate the new.
> > > 
> > > However, CPR can simply send the file descriptors to new QEMU,
> > > with no special management actions required.  The user enables
> > > this behavior by specifing '-netdev tap,cpr=on'.  The default
> > > is cpr=off.
> > > 
> > > Steve Sistare (8):
> > >    migration: stop vm earlier for cpr
> > >    migration: cpr setup notifier
> > >    vhost: reset vhost devices for cpr
> > >    cpr: delete all fds
> > >    Revert "vhost-backend: remove vhost_kernel_reset_device()"
> > >    tap: common return label
> > >    tap: cpr support
> > >    tap: postload fix for cpr
> > > 
> > >   qapi/net.json             |   5 +-
> > >   include/hw/virtio/vhost.h |   1 +
> > >   include/migration/cpr.h   |   3 +-
> > >   include/net/tap.h         |   1 +
> > >   hw/net/virtio-net.c       |  20 +++++++
> > >   hw/vfio/device.c          |   2 +-
> > >   hw/virtio/vhost-backend.c |   6 ++
> > >   hw/virtio/vhost.c         |  32 +++++++++++
> > >   migration/cpr.c           |  24 ++++++--
> > >   migration/migration.c     |  38 ++++++++-----
> > >   net/tap-win32.c           |   5 ++
> > >   net/tap.c                 | 141 +++++++++++++++++++++++++++++++++++-----------
> > >   12 files changed, 223 insertions(+), 55 deletions(-)
> > 
> > Hi Steve,
> > 
> > Patches 1-2 seem to potentially interact with your arm pending
> > interrupts fix. Do we want them together?
> 
> Good observation, thanks!.  I may need patches 1-2 to completely close
> the dropped interrupt race.  I will do more testing to verify that.

Sorry to respond late.. Could I request (for each of the patches 1 & 2)
in-code comments explaining the order of events?

For example, patch 1 moved stop_vm even earlier for CPR.  It used to be
early because it wants to avoid dirty tracking: this is something I didn't
realize but remembered after re-read the doc..  Now it further needs to
avoid the notifiers.  A comment above stop_vm for cpr explaining all these
order of events would be really helpful (including any necessary doc update).

-- 
Peter Xu

Re: [RFC V2 0/8] Live update: tap and vhost

[Steven Sistare]

On 8/5/2025 3:53 PM, Steven Sistare wrote:
> On 8/5/2025 9:54 AM, Fabiano Rosas wrote:
>> Steve Sistare <steven.sistare@oracle.com> writes:
>>
>>> Tap and vhost devices can be preserved during cpr-transfer using
>>> traditional live migration methods, wherein the management layer
>>> creates new interfaces for the target and fiddles with 'ip link'
>>> to deactivate the old interface and activate the new.
>>>
>>> However, CPR can simply send the file descriptors to new QEMU,
>>> with no special management actions required.  The user enables
>>> this behavior by specifing '-netdev tap,cpr=on'.  The default
>>> is cpr=off.
>>>
>>> Steve Sistare (8):
>>>    migration: stop vm earlier for cpr
>>>    migration: cpr setup notifier
>>>    vhost: reset vhost devices for cpr
>>>    cpr: delete all fds
>>>    Revert "vhost-backend: remove vhost_kernel_reset_device()"
>>>    tap: common return label
>>>    tap: cpr support
>>>    tap: postload fix for cpr
>>>
>>>   qapi/net.json             |   5 +-
>>>   include/hw/virtio/vhost.h |   1 +
>>>   include/migration/cpr.h   |   3 +-
>>>   include/net/tap.h         |   1 +
>>>   hw/net/virtio-net.c       |  20 +++++++
>>>   hw/vfio/device.c          |   2 +-
>>>   hw/virtio/vhost-backend.c |   6 ++
>>>   hw/virtio/vhost.c         |  32 +++++++++++
>>>   migration/cpr.c           |  24 ++++++--
>>>   migration/migration.c     |  38 ++++++++-----
>>>   net/tap-win32.c           |   5 ++
>>>   net/tap.c                 | 141 +++++++++++++++++++++++++++++++++++-----------
>>>   12 files changed, 223 insertions(+), 55 deletions(-)
>>
>> Hi Steve,
>>
>> Patches 1-2 seem to potentially interact with your arm pending
>> interrupts fix. Do we want them together?
> 
> Good observation, thanks!.  I may need patches 1-2 to completely close
> the dropped interrupt race.  I will do more testing to verify that.

aarch64 does not need patches 1-2 to fix interrupts.
It only relies on MIG_EVENT_PRECOPY_DONE, whose relative ordering is not
affected by patches 1-2.

So, the arm patch gicv3 V3 can be pulled.

I am still looking at patches 1-2 wrt x86 interrupts.

- Steve

Re: [RFC V2 0/8] Live update: tap and vhost

[Vladimir Sementsov-Ogievskiy]

On 17.07.25 21:39, Steve Sistare wrote:
> Tap and vhost devices can be preserved during cpr-transfer using
> traditional live migration methods, wherein the management layer
> creates new interfaces for the target and fiddles with 'ip link'
> to deactivate the old interface and activate the new.
> 
> However, CPR can simply send the file descriptors to new QEMU,
> with no special management actions required.  The user enables
> this behavior by specifing '-netdev tap,cpr=on'.  The default
> is cpr=off.
> 


Hi Steve!

First, me trying to test the series:

SOURCE:

sudo build/qemu-system-x86_64 -display none -vga none -device pxb-pcie,bus_nr=128,bus=pcie.0,id=pcie.1 -device pcie-root-port,id=s0,slot=0,bus=pcie.1 -device pcie-root-port,id=s1,slot=1,bus=pcie.1 -device pcie-root-port,id=s2,slot=2,bus=pcie.1 -hda /home/vsementsov/work/vms/newfocal.raw -m 4G -enable-kvm -M q35 -vnc :0 -nodefaults -vga std -qmp stdio -msg timestamp -S -object memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on -machine memory-backend=ram0 -machine aux-ram-share=on

{"execute": "qmp_capabilities"}
{"return": {}}
{"execute": "netdev_add", "arguments": {"cpr": true, "script": "no", "downscript": "no", "vhostforce": false, "vhost": false, "queues": 4, "ifname": "tap0", "type": "tap", "id": "netdev.1"}}
{"return": {}}
{"execute": "device_add", "arguments": {"disable-legacy": "off", "bus": "s1", "netdev": "netdev.1", "driver": "virtio-net-pci", "vectors": 18, "mq": true, "romfile": "", "mac": "d6:0d:75:f8:0f:b7", "id": "vnet.1"}}
{"return": {}}
{"execute": "cont"}
{"timestamp": {"seconds": 1755977653, "microseconds": 248749}, "event": "RESUME"}
{"return": {}}
{"timestamp": {"seconds": 1755977657, "microseconds": 366274}, "event": "NIC_RX_FILTER_CHANGED", "data": {"name": "vnet.1", "path": "/machine/peripheral/vnet.1/virtio-backend"}}
{"execute": "migrate-set-parameters", "arguments": {"mode": "cpr-transfer"}}
{"return": {}}
{"execute": "migrate", "arguments": {"channels": [{"channel-type": "main", "addr": {"path": "/tmp/migr.sock", "transport": "socket", "type": "unix"}}, {"channel-type": "cpr", "addr": {"path": "/tmp/cpr.sock", "transport": "socket", "type": "unix"}}]}}
{"timestamp": {"seconds": 1755977767, "microseconds": 835571}, "event": "STOP"}
{"return": {}}


TARGET:

sudo build/qemu-system-x86_64 -display none -vga none -device pxb-pcie,bus_nr=128,bus=pcie.0,id=pcie.1 -device pcie-root-port,id=s0,slot=0,bus=pcie.1 -device pcie-root-port,id=s1,slot=1,bus=pcie.1 -device pcie-root-port,id=s2,slot=2,bus=pcie.1 -hda /home/vsementsov/work/vms/newfocal.raw -m 4G -enable-kvm -M q35 -vnc :1 -nodefaults -vga std -qmp stdio -S -object memory-backend-file,id=ram0,size=4G,mem-p
ath=/dev/shm/ram0,share=on -machine memory-backend=ram0 -machine aux-ram-share=on -incoming defer -incoming '{"channel-type": "cpr","addr": { "transport": "socket","type": "unix", "path": "/tmp/cpr.sock"}}'

<need to wait until "migrate" on source>

{"execute": "qmp_capabilities"}
{"return": {}}
{"execute": "netdev_add", "arguments": {"cpr": true, "script": "no", "downscript": "no", "vhostforce": false, "vhost": false, "queues": 4, "ifname": "tap0", "type": "tap", "id": "netdev.1"}}
{"return": {}}
{"execute": "device_add", "arguments": {"disable-legacy": "off", "bus": "s1", "netdev": "netdev.1", "driver": "virtio-net-pci", "vectors": 18, "mq": true, "romfile": "", "mac": "d6:0d:75:f8:0f:b7", "id": "vnet.1"}}
could not disable queue
qemu-system-x86_64: ../hw/net/virtio-net.c:771: virtio_net_set_queue_pairs: Assertion `!r' failed.
fish: Job 1, 'sudo build/qemu-system-x86_64 -…' terminated by signal SIGABRT (Abort)


So, it crashes on device_add..


Second, I've come a long way, backporting you TAP v1 series together with needed parts of CPR and migration channels to QEMU 7.2, fixing different issues (like, avoid reinitialization of vnet_hdr length on target, avoid simultaneous use of tap on source an target, avoid making the fd blocking again on target), and it finally started to work.

But next, I went to support similar migration for vhost-user-blk, and that was a lot more complex. No reason to pass an fd in preliminary stage, when source is running (like in CPR), because:

1. we just can't use the fd on target at all, until we stop use it on source, otherwise we just break vhost-user-blk protocol on the wire (unlike TAP, where some ioctls called on target doesn't break source)
2. we have to pass enough additional variables, which are simpler to pass through normal migration channel (how to pass anything except fds through cpr channel?)

So, I decided to go another way, and just migrate everything backend-related including fds through main migration channel. Of course, this requires deep reworking of device initialization in case of incoming migration (but for vhost-user-blk we need it anyway). The feature is in my series "[PATCH 00/33] vhost-user-blk: live-backend local migration" (you are in CC).

The success with vhost-user-blk (of-course) make me rethink TAP migration too: try to avoid using additional cpr channel and unusual waiting for QMP interface on target. And, I've just sent an RFC: "[RFC 0/7] virtio-net: live-TAP local migration"

What do you think?

-- 
Best regards,
Vladimir

Re: [RFC V2 0/8] Live update: tap and vhost

[Steven Sistare]

On 8/23/2025 5:53 PM, Vladimir Sementsov-Ogievskiy wrote:
> On 17.07.25 21:39, Steve Sistare wrote:
>> Tap and vhost devices can be preserved during cpr-transfer using
>> traditional live migration methods, wherein the management layer
>> creates new interfaces for the target and fiddles with 'ip link'
>> to deactivate the old interface and activate the new.
>>
>> However, CPR can simply send the file descriptors to new QEMU,
>> with no special management actions required.  The user enables
>> this behavior by specifing '-netdev tap,cpr=on'.  The default
>> is cpr=off.
> 
> Hi Steve!
> 
> First, me trying to test the series:

Thank-you Vladimir for all the work you are doing in this area.  I have
reproduced the "virtio_net_set_queue_pairs: Assertion `!r' failed." bug.
Let me dig into that before I study the larger questions you pose
about preserving tap/vhost-user-blk in local migration versus cpr.

- Steve

> SOURCE:
> 
> sudo build/qemu-system-x86_64 -display none -vga none -device pxb-pcie,bus_nr=128,bus=pcie.0,id=pcie.1 -device pcie-root-port,id=s0,slot=0,bus=pcie.1 -device pcie-root-port,id=s1,slot=1,bus=pcie.1 -device pcie-root-port,id=s2,slot=2,bus=pcie.1 -hda /home/vsementsov/work/vms/newfocal.raw -m 4G -enable-kvm -M q35 -vnc :0 -nodefaults -vga std -qmp stdio -msg timestamp -S -object memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on -machine memory-backend=ram0 -machine aux-ram-share=on
> 
> {"execute": "qmp_capabilities"}
> {"return": {}}
> {"execute": "netdev_add", "arguments": {"cpr": true, "script": "no", "downscript": "no", "vhostforce": false, "vhost": false, "queues": 4, "ifname": "tap0", "type": "tap", "id": "netdev.1"}}
> {"return": {}}
> {"execute": "device_add", "arguments": {"disable-legacy": "off", "bus": "s1", "netdev": "netdev.1", "driver": "virtio-net-pci", "vectors": 18, "mq": true, "romfile": "", "mac": "d6:0d:75:f8:0f:b7", "id": "vnet.1"}}
> {"return": {}}
> {"execute": "cont"}
> {"timestamp": {"seconds": 1755977653, "microseconds": 248749}, "event": "RESUME"}
> {"return": {}}
> {"timestamp": {"seconds": 1755977657, "microseconds": 366274}, "event": "NIC_RX_FILTER_CHANGED", "data": {"name": "vnet.1", "path": "/machine/peripheral/vnet.1/virtio-backend"}}
> {"execute": "migrate-set-parameters", "arguments": {"mode": "cpr-transfer"}}
> {"return": {}}
> {"execute": "migrate", "arguments": {"channels": [{"channel-type": "main", "addr": {"path": "/tmp/migr.sock", "transport": "socket", "type": "unix"}}, {"channel-type": "cpr", "addr": {"path": "/tmp/cpr.sock", "transport": "socket", "type": "unix"}}]}}
> {"timestamp": {"seconds": 1755977767, "microseconds": 835571}, "event": "STOP"}
> {"return": {}}
> 
> 
> TARGET:
> 
> sudo build/qemu-system-x86_64 -display none -vga none -device pxb-pcie,bus_nr=128,bus=pcie.0,id=pcie.1 -device pcie-root-port,id=s0,slot=0,bus=pcie.1 -device pcie-root-port,id=s1,slot=1,bus=pcie.1 -device pcie-root-port,id=s2,slot=2,bus=pcie.1 -hda /home/vsementsov/work/vms/newfocal.raw -m 4G -enable-kvm -M q35 -vnc :1 -nodefaults -vga std -qmp stdio -S -object memory-backend-file,id=ram0,size=4G,mem-p
> ath=/dev/shm/ram0,share=on -machine memory-backend=ram0 -machine aux-ram-share=on -incoming defer -incoming '{"channel-type": "cpr","addr": { "transport": "socket","type": "unix", "path": "/tmp/cpr.sock"}}'
> 
> <need to wait until "migrate" on source>
> 
> {"execute": "qmp_capabilities"}
> {"return": {}}
> {"execute": "netdev_add", "arguments": {"cpr": true, "script": "no", "downscript": "no", "vhostforce": false, "vhost": false, "queues": 4, "ifname": "tap0", "type": "tap", "id": "netdev.1"}}
> {"return": {}}
> {"execute": "device_add", "arguments": {"disable-legacy": "off", "bus": "s1", "netdev": "netdev.1", "driver": "virtio-net-pci", "vectors": 18, "mq": true, "romfile": "", "mac": "d6:0d:75:f8:0f:b7", "id": "vnet.1"}}
> could not disable queue
> qemu-system-x86_64: ../hw/net/virtio-net.c:771: virtio_net_set_queue_pairs: Assertion `!r' failed.
> fish: Job 1, 'sudo build/qemu-system-x86_64 -…' terminated by signal SIGABRT (Abort)
> 
> 
> So, it crashes on device_add..
> 
> 
> Second, I've come a long way, backporting you TAP v1 series together with needed parts of CPR and migration channels to QEMU 7.2, fixing different issues (like, avoid reinitialization of vnet_hdr length on target, avoid simultaneous use of tap on source an target, avoid making the fd blocking again on target), and it finally started to work.
> 
> But next, I went to support similar migration for vhost-user-blk, and that was a lot more complex. No reason to pass an fd in preliminary stage, when source is running (like in CPR), because:
> 
> 1. we just can't use the fd on target at all, until we stop use it on source, otherwise we just break vhost-user-blk protocol on the wire (unlike TAP, where some ioctls called on target doesn't break source)
> 2. we have to pass enough additional variables, which are simpler to pass through normal migration channel (how to pass anything except fds through cpr channel?)
> 
> So, I decided to go another way, and just migrate everything backend-related including fds through main migration channel. Of course, this requires deep reworking of device initialization in case of incoming migration (but for vhost-user-blk we need it anyway). The feature is in my series "[PATCH 00/33] vhost-user-blk: live-backend local migration" (you are in CC).
> 
> The success with vhost-user-blk (of-course) make me rethink TAP migration too: try to avoid using additional cpr channel and unusual waiting for QMP interface on target. And, I've just sent an RFC: "[RFC 0/7] virtio-net: live-TAP local migration"
> 
> What do you think?
> 

Re: [RFC V2 0/8] Live update: tap and vhost

[Steven Sistare]

[-- Attachment #1: Type: text/plain, Size: 6579 bytes --]

On 8/28/2025 11:48 AM, Steven Sistare wrote:
> On 8/23/2025 5:53 PM, Vladimir Sementsov-Ogievskiy wrote:
>> On 17.07.25 21:39, Steve Sistare wrote:
>>> Tap and vhost devices can be preserved during cpr-transfer using
>>> traditional live migration methods, wherein the management layer
>>> creates new interfaces for the target and fiddles with 'ip link'
>>> to deactivate the old interface and activate the new.
>>>
>>> However, CPR can simply send the file descriptors to new QEMU,
>>> with no special management actions required.  The user enables
>>> this behavior by specifing '-netdev tap,cpr=on'.  The default
>>> is cpr=off.
>>
>> Hi Steve!
>>
>> First, me trying to test the series:
> 
> Thank-you Vladimir for all the work you are doing in this area.  I have
> reproduced the "virtio_net_set_queue_pairs: Assertion `!r' failed." bug.
> Let me dig into that before I study the larger questions you pose
> about preserving tap/vhost-user-blk in local migration versus cpr.

I have reproduced your journey!  I fixed the assertion, the vnet_hdr, and
the blocking fd problems which you allude to.  The attached patch fixes
them, and will be squashed into the series.

Ben, you also reported the !r assertion failure, so this fix should help
you also.

>> SOURCE:
>>
>> sudo build/qemu-system-x86_64 -display none -vga none -device pxb-pcie,bus_nr=128,bus=pcie.0,id=pcie.1 -device pcie-root-port,id=s0,slot=0,bus=pcie.1 -device pcie-root-port,id=s1,slot=1,bus=pcie.1 -device pcie-root-port,id=s2,slot=2,bus=pcie.1 -hda /home/vsementsov/work/vms/newfocal.raw -m 4G -enable-kvm -M q35 -vnc :0 -nodefaults -vga std -qmp stdio -msg timestamp -S -object memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on -machine memory-backend=ram0 -machine aux-ram-share=on
>>
>> {"execute": "qmp_capabilities"}
>> {"return": {}}
>> {"execute": "netdev_add", "arguments": {"cpr": true, "script": "no", "downscript": "no", "vhostforce": false, "vhost": false, "queues": 4, "ifname": "tap0", "type": "tap", "id": "netdev.1"}}
>> {"return": {}}
>> {"execute": "device_add", "arguments": {"disable-legacy": "off", "bus": "s1", "netdev": "netdev.1", "driver": "virtio-net-pci", "vectors": 18, "mq": true, "romfile": "", "mac": "d6:0d:75:f8:0f:b7", "id": "vnet.1"}}
>> {"return": {}}
>> {"execute": "cont"}
>> {"timestamp": {"seconds": 1755977653, "microseconds": 248749}, "event": "RESUME"}
>> {"return": {}}
>> {"timestamp": {"seconds": 1755977657, "microseconds": 366274}, "event": "NIC_RX_FILTER_CHANGED", "data": {"name": "vnet.1", "path": "/machine/peripheral/vnet.1/virtio-backend"}}
>> {"execute": "migrate-set-parameters", "arguments": {"mode": "cpr-transfer"}}
>> {"return": {}}
>> {"execute": "migrate", "arguments": {"channels": [{"channel-type": "main", "addr": {"path": "/tmp/migr.sock", "transport": "socket", "type": "unix"}}, {"channel-type": "cpr", "addr": {"path": "/tmp/cpr.sock", "transport": "socket", "type": "unix"}}]}}
>> {"timestamp": {"seconds": 1755977767, "microseconds": 835571}, "event": "STOP"}
>> {"return": {}}
>>
>> TARGET:
>>
>> sudo build/qemu-system-x86_64 -display none -vga none -device pxb-pcie,bus_nr=128,bus=pcie.0,id=pcie.1 -device pcie-root-port,id=s0,slot=0,bus=pcie.1 -device pcie-root-port,id=s1,slot=1,bus=pcie.1 -device pcie-root-port,id=s2,slot=2,bus=pcie.1 -hda /home/vsementsov/work/vms/newfocal.raw -m 4G -enable-kvm -M q35 -vnc :1 -nodefaults -vga std -qmp stdio -S -object memory-backend-file,id=ram0,size=4G,mem-p
>> ath=/dev/shm/ram0,share=on -machine memory-backend=ram0 -machine aux-ram-share=on -incoming defer -incoming '{"channel-type": "cpr","addr": { "transport": "socket","type": "unix", "path": "/tmp/cpr.sock"}}'
>>
>> <need to wait until "migrate" on source>
>>
>> {"execute": "qmp_capabilities"}
>> {"return": {}}
>> {"execute": "netdev_add", "arguments": {"cpr": true, "script": "no", "downscript": "no", "vhostforce": false, "vhost": false, "queues": 4, "ifname": "tap0", "type": "tap", "id": "netdev.1"}}
>> {"return": {}}
>> {"execute": "device_add", "arguments": {"disable-legacy": "off", "bus": "s1", "netdev": "netdev.1", "driver": "virtio-net-pci", "vectors": 18, "mq": true, "romfile": "", "mac": "d6:0d:75:f8:0f:b7", "id": "vnet.1"}}
>> could not disable queue
>> qemu-system-x86_64: ../hw/net/virtio-net.c:771: virtio_net_set_queue_pairs: Assertion `!r' failed.
>> fish: Job 1, 'sudo build/qemu-system-x86_64 -…' terminated by signal SIGABRT (Abort)
>>
>> So, it crashes on device_add..
>>
>> Second, I've come a long way, backporting you TAP v1 series together with needed parts of CPR and migration channels to QEMU 7.2, fixing different issues (like, avoid reinitialization of vnet_hdr length on target, avoid simultaneous use of tap on source an target, avoid making the fd blocking again on target), and it finally started to work.
>>
>> But next, I went to support similar migration for vhost-user-blk, and that was a lot more complex. No reason to pass an fd in preliminary stage, when source is running (like in CPR), because:
>>
>> 1. we just can't use the fd on target at all, until we stop use it on source, otherwise we just break vhost-user-blk protocol on the wire (unlike TAP, where some ioctls called on target doesn't break source)
>> 2. we have to pass enough additional variables, which are simpler to pass through normal migration channel (how to pass anything except fds through cpr channel?)

You can pass extra state through the cpr channel.  See for example vmstate_cpr_vfio_device,
and how vmstate_cpr_vfio_devices is defined as a sub-section of vmstate_cpr_state.

>> So, I decided to go another way, and just migrate everything backend-related including fds through main migration channel. Of course, this requires deep reworking of device initialization in case of incoming migration (but for vhost-user-blk we need it anyway). The feature is in my series "[PATCH 00/33] vhost-user-blk: live-backend local migration" (you are in CC).

You did a lot of work in those series!
I suspect much less rework of initialization is required if you pass variables in cpr state.

>> The success with vhost-user-blk (of-course) make me rethink TAP migration too: try to avoid using additional cpr channel and unusual waiting for QMP interface on target. And, I've just sent an RFC: "[RFC 0/7] virtio-net: live-TAP local migration"

Is there a use case for this outside of CPR?
CPR is intended to be the "local migration" solution that does it all :)
But if you do proceed with your local migration tap solution, I would want
to see that CPR could also use your code paths.

- Steve

[-- Attachment #2: 0001-tap-cpr-fixes.patch --]
[-- Type: text/plain, Size: 2755 bytes --]

From 191521210222638940c17d4daefc313d4ad61aa3 Mon Sep 17 00:00:00 2001
From: Steve Sistare <steven.sistare@oracle.com>
Date: Thu, 28 Aug 2025 13:47:15 -0700
Subject: [PATCH] tap: cpr fixes

Fix "virtio_net_set_queue_pairs: Assertion `!r' failed."
Fix "virtio-net: saved image requires vnet_hdr=on"
Do not change blocking mode of incoming cpr fd's.

Reported-by: Ben Chaney <bchaney@akamai.com>
Reported-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>

Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
---
 hw/net/virtio-net.c | 6 ++++++
 io/channel-socket.c | 5 ++++-
 net/tap.c           | 2 ++
 stubs/cpr.c         | 5 +++++
 4 files changed, 17 insertions(+), 1 deletion(-)

diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index 7c40cca..7dd8a80 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -37,6 +37,7 @@
 #include "qapi/qapi-types-migration.h"
 #include "qapi/qapi-events-migration.h"
 #include "hw/virtio/virtio-access.h"
+#include "migration/cpr.h"
 #include "migration/misc.h"
 #include "standard-headers/linux/ethtool.h"
 #include "system/system.h"
@@ -758,6 +759,11 @@ static void virtio_net_set_queue_pairs(VirtIONet *n)
     int i;
     int r;
 
+    if (cpr_is_incoming()) {
+        /* peers are already attached, do nothing */
+        return;
+    }
+
     if (n->nic->peer_deleted) {
         return;
     }
diff --git a/io/channel-socket.c b/io/channel-socket.c
index 3b7ca92..736d39d 100644
--- a/io/channel-socket.c
+++ b/io/channel-socket.c
@@ -24,6 +24,7 @@
 #include "io/channel-socket.h"
 #include "io/channel-util.h"
 #include "io/channel-watch.h"
+#include "migration/cpr.h"
 #include "trace.h"
 #include "qapi/clone-visitor.h"
 #ifdef CONFIG_LINUX
@@ -498,7 +499,9 @@ static void qio_channel_socket_copy_fds(struct msghdr *msg,
             }
 
             /* O_NONBLOCK is preserved across SCM_RIGHTS so reset it */
-            qemu_socket_set_block(fd);
+            if (!cpr_is_incoming()) {
+                qemu_socket_set_block(fd);
+            }
 
 #ifndef MSG_CMSG_CLOEXEC
             qemu_set_cloexec(fd);
diff --git a/net/tap.c b/net/tap.c
index 25ff96f..f95ffe9 100644
--- a/net/tap.c
+++ b/net/tap.c
@@ -1042,6 +1042,8 @@ free_fail:
                 if (cpr && fd >= 0) {
                     cpr_save_fd(name, TAP_FD_INDEX(i), fd);
                 }
+            } else {
+                vnet_hdr = tap->has_vnet_hdr ? tap->vnet_hdr : 1;
             }
             if (fd == -1) {
                 ret = -1;
diff --git a/stubs/cpr.c b/stubs/cpr.c
index 6a5c320..86a507c 100644
--- a/stubs/cpr.c
+++ b/stubs/cpr.c
@@ -19,3 +19,8 @@ int cpr_find_fd(const char *name, int id)
 {
     return -1;
 }
+
+bool cpr_is_incoming(void)
+{
+    return false;
+}
-- 
1.8.3.1

Re: [RFC V2 0/8] Live update: tap and vhost

[Vladimir Sementsov-Ogievskiy]

On 29.08.25 22:37, Steven Sistare wrote:
> On 8/28/2025 11:48 AM, Steven Sistare wrote:
>> On 8/23/2025 5:53 PM, Vladimir Sementsov-Ogievskiy wrote:
>>> On 17.07.25 21:39, Steve Sistare wrote:
>>>> Tap and vhost devices can be preserved during cpr-transfer using
>>>> traditional live migration methods, wherein the management layer
>>>> creates new interfaces for the target and fiddles with 'ip link'
>>>> to deactivate the old interface and activate the new.
>>>>
>>>> However, CPR can simply send the file descriptors to new QEMU,
>>>> with no special management actions required.  The user enables
>>>> this behavior by specifing '-netdev tap,cpr=on'.  The default
>>>> is cpr=off.
>>>
>>> Hi Steve!
>>>
>>> First, me trying to test the series:
>>
>> Thank-you Vladimir for all the work you are doing in this area.  I have
>> reproduced the "virtio_net_set_queue_pairs: Assertion `!r' failed." bug.
>> Let me dig into that before I study the larger questions you pose
>> about preserving tap/vhost-user-blk in local migration versus cpr.
> 
> I have reproduced your journey!  I fixed the assertion, the vnet_hdr, and
> the blocking fd problems which you allude to.  The attached patch fixes
> them, and will be squashed into the series.
> 
> Ben, you also reported the !r assertion failure, so this fix should help
> you also.
> 
>>> SOURCE:
>>>
>>> sudo build/qemu-system-x86_64 -display none -vga none -device pxb-pcie,bus_nr=128,bus=pcie.0,id=pcie.1 -device pcie-root-port,id=s0,slot=0,bus=pcie.1 -device pcie-root-port,id=s1,slot=1,bus=pcie.1 -device pcie-root-port,id=s2,slot=2,bus=pcie.1 -hda /home/vsementsov/work/vms/newfocal.raw -m 4G -enable-kvm -M q35 -vnc :0 -nodefaults -vga std -qmp stdio -msg timestamp -S -object memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on -machine memory-backend=ram0 -machine aux-ram-share=on
>>>
>>> {"execute": "qmp_capabilities"}
>>> {"return": {}}
>>> {"execute": "netdev_add", "arguments": {"cpr": true, "script": "no", "downscript": "no", "vhostforce": false, "vhost": false, "queues": 4, "ifname": "tap0", "type": "tap", "id": "netdev.1"}}
>>> {"return": {}}
>>> {"execute": "device_add", "arguments": {"disable-legacy": "off", "bus": "s1", "netdev": "netdev.1", "driver": "virtio-net-pci", "vectors": 18, "mq": true, "romfile": "", "mac": "d6:0d:75:f8:0f:b7", "id": "vnet.1"}}
>>> {"return": {}}
>>> {"execute": "cont"}
>>> {"timestamp": {"seconds": 1755977653, "microseconds": 248749}, "event": "RESUME"}
>>> {"return": {}}
>>> {"timestamp": {"seconds": 1755977657, "microseconds": 366274}, "event": "NIC_RX_FILTER_CHANGED", "data": {"name": "vnet.1", "path": "/machine/peripheral/vnet.1/virtio-backend"}}
>>> {"execute": "migrate-set-parameters", "arguments": {"mode": "cpr-transfer"}}
>>> {"return": {}}
>>> {"execute": "migrate", "arguments": {"channels": [{"channel-type": "main", "addr": {"path": "/tmp/migr.sock", "transport": "socket", "type": "unix"}}, {"channel-type": "cpr", "addr": {"path": "/tmp/cpr.sock", "transport": "socket", "type": "unix"}}]}}
>>> {"timestamp": {"seconds": 1755977767, "microseconds": 835571}, "event": "STOP"}
>>> {"return": {}}
>>>
>>> TARGET:
>>>
>>> sudo build/qemu-system-x86_64 -display none -vga none -device pxb-pcie,bus_nr=128,bus=pcie.0,id=pcie.1 -device pcie-root-port,id=s0,slot=0,bus=pcie.1 -device pcie-root-port,id=s1,slot=1,bus=pcie.1 -device pcie-root-port,id=s2,slot=2,bus=pcie.1 -hda /home/vsementsov/work/vms/newfocal.raw -m 4G -enable-kvm -M q35 -vnc :1 -nodefaults -vga std -qmp stdio -S -object memory-backend-file,id=ram0,size=4G,mem-p
>>> ath=/dev/shm/ram0,share=on -machine memory-backend=ram0 -machine aux-ram-share=on -incoming defer -incoming '{"channel-type": "cpr","addr": { "transport": "socket","type": "unix", "path": "/tmp/cpr.sock"}}'
>>>
>>> <need to wait until "migrate" on source>
>>>
>>> {"execute": "qmp_capabilities"}
>>> {"return": {}}
>>> {"execute": "netdev_add", "arguments": {"cpr": true, "script": "no", "downscript": "no", "vhostforce": false, "vhost": false, "queues": 4, "ifname": "tap0", "type": "tap", "id": "netdev.1"}}
>>> {"return": {}}
>>> {"execute": "device_add", "arguments": {"disable-legacy": "off", "bus": "s1", "netdev": "netdev.1", "driver": "virtio-net-pci", "vectors": 18, "mq": true, "romfile": "", "mac": "d6:0d:75:f8:0f:b7", "id": "vnet.1"}}
>>> could not disable queue
>>> qemu-system-x86_64: ../hw/net/virtio-net.c:771: virtio_net_set_queue_pairs: Assertion `!r' failed.
>>> fish: Job 1, 'sudo build/qemu-system-x86_64 -…' terminated by signal SIGABRT (Abort)
>>>
>>> So, it crashes on device_add..
>>>
>>> Second, I've come a long way, backporting you TAP v1 series together with needed parts of CPR and migration channels to QEMU 7.2, fixing different issues (like, avoid reinitialization of vnet_hdr length on target, avoid simultaneous use of tap on source an target, avoid making the fd blocking again on target), and it finally started to work.
>>>
>>> But next, I went to support similar migration for vhost-user-blk, and that was a lot more complex. No reason to pass an fd in preliminary stage, when source is running (like in CPR), because:
>>>
>>> 1. we just can't use the fd on target at all, until we stop use it on source, otherwise we just break vhost-user-blk protocol on the wire (unlike TAP, where some ioctls called on target doesn't break source)
>>> 2. we have to pass enough additional variables, which are simpler to pass through normal migration channel (how to pass anything except fds through cpr channel?)
> 
> You can pass extra state through the cpr channel.  See for example vmstate_cpr_vfio_device,
> and how vmstate_cpr_vfio_devices is defined as a sub-section of vmstate_cpr_state.

O, I missed this.

Hmm. Still, finally CPR becomes just an additional stage of migration, which is done prior device initialization on target..

Didn't you think of integrating it to the common scheme: so that devices may have .vmsd_cpr in addition to .vmsd? This way we don't need a global CPR state, and CPR stage of migration will work the same way as normal migration?

Still2, if we pass some state in CPR it should be a kind of constant. We need a guarantee that it will not change between migration start and source stop.

> 
>>> So, I decided to go another way, and just migrate everything backend-related including fds through main migration channel. Of course, this requires deep reworking of device initialization in case of incoming migration (but for vhost-user-blk we need it anyway). The feature is in my series "[PATCH 00/33] vhost-user-blk: live-backend local migration" (you are in CC).
> 
> You did a lot of work in those series!
> I suspect much less rework of initialization is required if you pass variables in cpr state.

Not sure. I had to rework initialization anyway, as initialization damaged the connection. And this lead me to idea "if rework anyway, why not to go with one migration channel".

> 
>>> The success with vhost-user-blk (of-course) make me rethink TAP migration too: try to avoid using additional cpr channel and unusual waiting for QMP interface on target. And, I've just sent an RFC: "[RFC 0/7] virtio-net: live-TAP local migration"
> 
> Is there a use case for this outside of CPR?

It just works without CPR.. Will CPR bring more benefit if I enable it in the setup with my local-tap + local-vhost-user-blk capabilities ( + ignore-shared of-course)?


> CPR is intended to be the "local migration" solution that does it all :)
> But if you do proceed with your local migration tap solution, I would want
> to see that CPR could also use your code paths.
> 
CPR can transparently use my code: you may enable both CPR and local-tap capability and it should work. Some devices will migrate their fds through CPR, TAP fds amd state will migrate through main migration channel. Making both channels to be unix-sockets should not be a considerable overhead I think.

Why I like my solution more:

- no additional channel
- no additional logic in management software (to handle target start with no QMP access until "migrate" command on source)
- less code to backport (that's personal, of course not an argument for final upstream solution)

It seems that CPR is simpler to support as we don't need to do deep rework of initialization code.. But in reality, there is a lot of work anyway: TAP, vhost-user-blk cases proves this. You series about vfio are also huge.

What is the benefit of CPR against simple (unix-socket) migration?


-- 
Best regards,
Vladimir

Re: [RFC V2 0/8] Live update: tap and vhost

[Steven Sistare]

On 9/1/2025 7:44 AM, Vladimir Sementsov-Ogievskiy wrote:
> On 29.08.25 22:37, Steven Sistare wrote:
>> On 8/28/2025 11:48 AM, Steven Sistare wrote:
>>> On 8/23/2025 5:53 PM, Vladimir Sementsov-Ogievskiy wrote:
>>>> On 17.07.25 21:39, Steve Sistare wrote:
>>>>> Tap and vhost devices can be preserved during cpr-transfer using
>>>>> traditional live migration methods, wherein the management layer
>>>>> creates new interfaces for the target and fiddles with 'ip link'
>>>>> to deactivate the old interface and activate the new.
>>>>>
>>>>> However, CPR can simply send the file descriptors to new QEMU,
>>>>> with no special management actions required.  The user enables
>>>>> this behavior by specifing '-netdev tap,cpr=on'.  The default
>>>>> is cpr=off.
>>>>
>>>> Hi Steve!
>>>>
>>>> First, me trying to test the series:
>>>
>>> Thank-you Vladimir for all the work you are doing in this area.  I have
>>> reproduced the "virtio_net_set_queue_pairs: Assertion `!r' failed." bug.
>>> Let me dig into that before I study the larger questions you pose
>>> about preserving tap/vhost-user-blk in local migration versus cpr.
>>
>> I have reproduced your journey!  I fixed the assertion, the vnet_hdr, and
>> the blocking fd problems which you allude to.  The attached patch fixes
>> them, and will be squashed into the series.
>>
>> Ben, you also reported the !r assertion failure, so this fix should help
>> you also.
>>
>>>> SOURCE:
>>>>
>>>> sudo build/qemu-system-x86_64 -display none -vga none -device pxb-pcie,bus_nr=128,bus=pcie.0,id=pcie.1 -device pcie-root-port,id=s0,slot=0,bus=pcie.1 -device pcie-root-port,id=s1,slot=1,bus=pcie.1 -device pcie-root-port,id=s2,slot=2,bus=pcie.1 -hda /home/vsementsov/work/vms/newfocal.raw -m 4G -enable-kvm -M q35 -vnc :0 -nodefaults -vga std -qmp stdio -msg timestamp -S -object memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on -machine memory-backend=ram0 -machine aux-ram-share=on
>>>>
>>>> {"execute": "qmp_capabilities"}
>>>> {"return": {}}
>>>> {"execute": "netdev_add", "arguments": {"cpr": true, "script": "no", "downscript": "no", "vhostforce": false, "vhost": false, "queues": 4, "ifname": "tap0", "type": "tap", "id": "netdev.1"}}
>>>> {"return": {}}
>>>> {"execute": "device_add", "arguments": {"disable-legacy": "off", "bus": "s1", "netdev": "netdev.1", "driver": "virtio-net-pci", "vectors": 18, "mq": true, "romfile": "", "mac": "d6:0d:75:f8:0f:b7", "id": "vnet.1"}}
>>>> {"return": {}}
>>>> {"execute": "cont"}
>>>> {"timestamp": {"seconds": 1755977653, "microseconds": 248749}, "event": "RESUME"}
>>>> {"return": {}}
>>>> {"timestamp": {"seconds": 1755977657, "microseconds": 366274}, "event": "NIC_RX_FILTER_CHANGED", "data": {"name": "vnet.1", "path": "/machine/peripheral/vnet.1/virtio-backend"}}
>>>> {"execute": "migrate-set-parameters", "arguments": {"mode": "cpr-transfer"}}
>>>> {"return": {}}
>>>> {"execute": "migrate", "arguments": {"channels": [{"channel-type": "main", "addr": {"path": "/tmp/migr.sock", "transport": "socket", "type": "unix"}}, {"channel-type": "cpr", "addr": {"path": "/tmp/cpr.sock", "transport": "socket", "type": "unix"}}]}}
>>>> {"timestamp": {"seconds": 1755977767, "microseconds": 835571}, "event": "STOP"}
>>>> {"return": {}}
>>>>
>>>> TARGET:
>>>>
>>>> sudo build/qemu-system-x86_64 -display none -vga none -device pxb-pcie,bus_nr=128,bus=pcie.0,id=pcie.1 -device pcie-root-port,id=s0,slot=0,bus=pcie.1 -device pcie-root-port,id=s1,slot=1,bus=pcie.1 -device pcie-root-port,id=s2,slot=2,bus=pcie.1 -hda /home/vsementsov/work/vms/newfocal.raw -m 4G -enable-kvm -M q35 -vnc :1 -nodefaults -vga std -qmp stdio -S -object memory-backend-file,id=ram0,size=4G,mem-p
>>>> ath=/dev/shm/ram0,share=on -machine memory-backend=ram0 -machine aux-ram-share=on -incoming defer -incoming '{"channel-type": "cpr","addr": { "transport": "socket","type": "unix", "path": "/tmp/cpr.sock"}}'
>>>>
>>>> <need to wait until "migrate" on source>
>>>>
>>>> {"execute": "qmp_capabilities"}
>>>> {"return": {}}
>>>> {"execute": "netdev_add", "arguments": {"cpr": true, "script": "no", "downscript": "no", "vhostforce": false, "vhost": false, "queues": 4, "ifname": "tap0", "type": "tap", "id": "netdev.1"}}
>>>> {"return": {}}
>>>> {"execute": "device_add", "arguments": {"disable-legacy": "off", "bus": "s1", "netdev": "netdev.1", "driver": "virtio-net-pci", "vectors": 18, "mq": true, "romfile": "", "mac": "d6:0d:75:f8:0f:b7", "id": "vnet.1"}}
>>>> could not disable queue
>>>> qemu-system-x86_64: ../hw/net/virtio-net.c:771: virtio_net_set_queue_pairs: Assertion `!r' failed.
>>>> fish: Job 1, 'sudo build/qemu-system-x86_64 -…' terminated by signal SIGABRT (Abort)
>>>>
>>>> So, it crashes on device_add..
>>>>
>>>> Second, I've come a long way, backporting you TAP v1 series together with needed parts of CPR and migration channels to QEMU 7.2, fixing different issues (like, avoid reinitialization of vnet_hdr length on target, avoid simultaneous use of tap on source an target, avoid making the fd blocking again on target), and it finally started to work.
>>>>
>>>> But next, I went to support similar migration for vhost-user-blk, and that was a lot more complex. No reason to pass an fd in preliminary stage, when source is running (like in CPR), because:
>>>>
>>>> 1. we just can't use the fd on target at all, until we stop use it on source, otherwise we just break vhost-user-blk protocol on the wire (unlike TAP, where some ioctls called on target doesn't break source)
>>>> 2. we have to pass enough additional variables, which are simpler to pass through normal migration channel (how to pass anything except fds through cpr channel?)
>>
>> You can pass extra state through the cpr channel.  See for example vmstate_cpr_vfio_device,
>> and how vmstate_cpr_vfio_devices is defined as a sub-section of vmstate_cpr_state.
> 
> O, I missed this.
> 
> Hmm. Still, finally CPR becomes just an additional stage of migration, which is done prior device initialization on target..
> 
> Didn't you think of integrating it to the common scheme: so that devices may have .vmsd_cpr in addition to .vmsd? This way we don't need a global CPR state, and CPR stage of migration will work the same way as normal migration?

I proposed a single migration stream containing pre-create state that was read early,
but that was rejected as too complex.

I also proposed refactoring initialization so the monitor and migration streams
could be opened earlier, but again rejected as too complex and/or not consistent with
a long term vision for reworking initialization.

> Still2, if we pass some state in CPR it should be a kind of constant. We need a guarantee that it will not change between migration start and source stop.
> 
>>
>>>> So, I decided to go another way, and just migrate everything backend-related including fds through main migration channel. Of course, this requires deep reworking of device initialization in case of incoming migration (but for vhost-user-blk we need it anyway). The feature is in my series "[PATCH 00/33] vhost-user-blk: live-backend local migration" (you are in CC).
>>
>> You did a lot of work in those series!
>> I suspect much less rework of initialization is required if you pass variables in cpr state.
> 
> Not sure. I had to rework initialization anyway, as initialization damaged the connection. And this lead me to idea "if rework anyway, why not to go with one migration channel".
> 
>>
>>>> The success with vhost-user-blk (of-course) make me rethink TAP migration too: try to avoid using additional cpr channel and unusual waiting for QMP interface on target. And, I've just sent an RFC: "[RFC 0/7] virtio-net: live-TAP local migration"
>>
>> Is there a use case for this outside of CPR?
> 
> It just works without CPR.. Will CPR bring more benefit if I enable it in the setup with my local-tap + local-vhost-user-blk capabilities ( + ignore-shared of-course)?
> 
> 
>> CPR is intended to be the "local migration" solution that does it all :)
>> But if you do proceed with your local migration tap solution, I would want
>> to see that CPR could also use your code paths.
>>
> CPR can transparently use my code: you may enable both CPR and local-tap capability and it should work. Some devices will migrate their fds through CPR, TAP fds amd state will migrate through main migration channel. 

OK, I believe that.

I also care about cpr-exec mode.  We use it internally, and I am trying to push
it upstream:
   https://lore.kernel.org/qemu-devel/1755191843-283480-8-git-send-email-steven.sistare@oracle.com/
I believe it would work with your code.  Migrated fd's in both the cpr channel and
the main migration channel would be handled differently as shown in vmstate-types.c
get_fd() and put_fd().  The fd is kept open across execv(), and vmstate represents
the fd by its value (eg a small integer), rather than as an object in the unix channel.

> Making both channels to be unix-sockets should not be a considerable overhead I think.
> 
> Why I like my solution more:
> 
> - no additional channel
> - no additional logic in management software (to handle target start with no QMP access until "migrate" command on source)
> - less code to backport (that's personal, of course not an argument for final upstream solution)
> 
> It seems that CPR is simpler to support as we don't need to do deep rework of initialization code.. But in reality, there is a lot of work anyway: TAP, vhost-user-blk cases proves this. You series about vfio are also huge.

TAP is the only case where we can compare both approaches, and the numbers tell
the story:

   TAP initialization refactoring: 277 insertions(+), 308 deletions(-)
   live-TAP local migration:       681 insertions(+), 72 deletions(-)
                         total:    958 insertions(+), 380 deletions(-)

   Live update tap and vhost:      223 insertions(+), 55 deletions(-)

For any given system, if the maintainers accept the larger amount of change,
then local migration is cool (and CPR made it possible by adding fd support
to vmstate+QEMUFile).  But the amount of change is a harder sell.
> What is the benefit of CPR against simple (unix-socket) migration?CPR supports vfio, iommufd, and pinned memory.  Memory backend objects are
created early, before the main migration stream is read, and squashing
CPR into migration for those cases would require a major change in how
qemu creates objects during live migration.

Hence CPR is the method that works for all types of objects.  The mgmt
layer does not need to support multiple methods of live update, depending
on what devices the VM contains.

- Steve

Re: [RFC V2 0/8] Live update: tap and vhost

[Vladimir Sementsov-Ogievskiy]

On 02.09.25 18:33, Steven Sistare wrote:
> On 9/1/2025 7:44 AM, Vladimir Sementsov-Ogievskiy wrote:
>> On 29.08.25 22:37, Steven Sistare wrote:
>>> On 8/28/2025 11:48 AM, Steven Sistare wrote:
>>>> On 8/23/2025 5:53 PM, Vladimir Sementsov-Ogievskiy wrote:
>>>>> On 17.07.25 21:39, Steve Sistare wrote:
>>>>>> Tap and vhost devices can be preserved during cpr-transfer using
>>>>>> traditional live migration methods, wherein the management layer
>>>>>> creates new interfaces for the target and fiddles with 'ip link'
>>>>>> to deactivate the old interface and activate the new.
>>>>>>
>>>>>> However, CPR can simply send the file descriptors to new QEMU,
>>>>>> with no special management actions required.  The user enables
>>>>>> this behavior by specifing '-netdev tap,cpr=on'.  The default
>>>>>> is cpr=off.
>>>>>
>>>>> Hi Steve!
>>>>>
>>>>> First, me trying to test the series:
>>>>
>>>> Thank-you Vladimir for all the work you are doing in this area.  I have
>>>> reproduced the "virtio_net_set_queue_pairs: Assertion `!r' failed." bug.
>>>> Let me dig into that before I study the larger questions you pose
>>>> about preserving tap/vhost-user-blk in local migration versus cpr.
>>>
>>> I have reproduced your journey!  I fixed the assertion, the vnet_hdr, and
>>> the blocking fd problems which you allude to.  The attached patch fixes
>>> them, and will be squashed into the series.
>>>
>>> Ben, you also reported the !r assertion failure, so this fix should help
>>> you also.
>>>
>>>>> SOURCE:
>>>>>
>>>>> sudo build/qemu-system-x86_64 -display none -vga none -device pxb-pcie,bus_nr=128,bus=pcie.0,id=pcie.1 -device pcie-root-port,id=s0,slot=0,bus=pcie.1 -device pcie-root-port,id=s1,slot=1,bus=pcie.1 -device pcie-root-port,id=s2,slot=2,bus=pcie.1 -hda /home/vsementsov/work/vms/newfocal.raw -m 4G -enable-kvm -M q35 -vnc :0 -nodefaults -vga std -qmp stdio -msg timestamp -S -object memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on -machine memory-backend=ram0 -machine aux-ram-share=on
>>>>>
>>>>> {"execute": "qmp_capabilities"}
>>>>> {"return": {}}
>>>>> {"execute": "netdev_add", "arguments": {"cpr": true, "script": "no", "downscript": "no", "vhostforce": false, "vhost": false, "queues": 4, "ifname": "tap0", "type": "tap", "id": "netdev.1"}}
>>>>> {"return": {}}
>>>>> {"execute": "device_add", "arguments": {"disable-legacy": "off", "bus": "s1", "netdev": "netdev.1", "driver": "virtio-net-pci", "vectors": 18, "mq": true, "romfile": "", "mac": "d6:0d:75:f8:0f:b7", "id": "vnet.1"}}
>>>>> {"return": {}}
>>>>> {"execute": "cont"}
>>>>> {"timestamp": {"seconds": 1755977653, "microseconds": 248749}, "event": "RESUME"}
>>>>> {"return": {}}
>>>>> {"timestamp": {"seconds": 1755977657, "microseconds": 366274}, "event": "NIC_RX_FILTER_CHANGED", "data": {"name": "vnet.1", "path": "/machine/peripheral/vnet.1/virtio-backend"}}
>>>>> {"execute": "migrate-set-parameters", "arguments": {"mode": "cpr-transfer"}}
>>>>> {"return": {}}
>>>>> {"execute": "migrate", "arguments": {"channels": [{"channel-type": "main", "addr": {"path": "/tmp/migr.sock", "transport": "socket", "type": "unix"}}, {"channel-type": "cpr", "addr": {"path": "/tmp/cpr.sock", "transport": "socket", "type": "unix"}}]}}
>>>>> {"timestamp": {"seconds": 1755977767, "microseconds": 835571}, "event": "STOP"}
>>>>> {"return": {}}
>>>>>
>>>>> TARGET:
>>>>>
>>>>> sudo build/qemu-system-x86_64 -display none -vga none -device pxb-pcie,bus_nr=128,bus=pcie.0,id=pcie.1 -device pcie-root-port,id=s0,slot=0,bus=pcie.1 -device pcie-root-port,id=s1,slot=1,bus=pcie.1 -device pcie-root-port,id=s2,slot=2,bus=pcie.1 -hda /home/vsementsov/work/vms/newfocal.raw -m 4G -enable-kvm -M q35 -vnc :1 -nodefaults -vga std -qmp stdio -S -object memory-backend-file,id=ram0,size=4G,mem-p
>>>>> ath=/dev/shm/ram0,share=on -machine memory-backend=ram0 -machine aux-ram-share=on -incoming defer -incoming '{"channel-type": "cpr","addr": { "transport": "socket","type": "unix", "path": "/tmp/cpr.sock"}}'
>>>>>
>>>>> <need to wait until "migrate" on source>
>>>>>
>>>>> {"execute": "qmp_capabilities"}
>>>>> {"return": {}}
>>>>> {"execute": "netdev_add", "arguments": {"cpr": true, "script": "no", "downscript": "no", "vhostforce": false, "vhost": false, "queues": 4, "ifname": "tap0", "type": "tap", "id": "netdev.1"}}
>>>>> {"return": {}}
>>>>> {"execute": "device_add", "arguments": {"disable-legacy": "off", "bus": "s1", "netdev": "netdev.1", "driver": "virtio-net-pci", "vectors": 18, "mq": true, "romfile": "", "mac": "d6:0d:75:f8:0f:b7", "id": "vnet.1"}}
>>>>> could not disable queue
>>>>> qemu-system-x86_64: ../hw/net/virtio-net.c:771: virtio_net_set_queue_pairs: Assertion `!r' failed.
>>>>> fish: Job 1, 'sudo build/qemu-system-x86_64 -…' terminated by signal SIGABRT (Abort)
>>>>>
>>>>> So, it crashes on device_add..
>>>>>
>>>>> Second, I've come a long way, backporting you TAP v1 series together with needed parts of CPR and migration channels to QEMU 7.2, fixing different issues (like, avoid reinitialization of vnet_hdr length on target, avoid simultaneous use of tap on source an target, avoid making the fd blocking again on target), and it finally started to work.
>>>>>
>>>>> But next, I went to support similar migration for vhost-user-blk, and that was a lot more complex. No reason to pass an fd in preliminary stage, when source is running (like in CPR), because:
>>>>>
>>>>> 1. we just can't use the fd on target at all, until we stop use it on source, otherwise we just break vhost-user-blk protocol on the wire (unlike TAP, where some ioctls called on target doesn't break source)
>>>>> 2. we have to pass enough additional variables, which are simpler to pass through normal migration channel (how to pass anything except fds through cpr channel?)
>>>
>>> You can pass extra state through the cpr channel.  See for example vmstate_cpr_vfio_device,
>>> and how vmstate_cpr_vfio_devices is defined as a sub-section of vmstate_cpr_state.
>>
>> O, I missed this.
>>
>> Hmm. Still, finally CPR becomes just an additional stage of migration, which is done prior device initialization on target..
>>
>> Didn't you think of integrating it to the common scheme: so that devices may have .vmsd_cpr in addition to .vmsd? This way we don't need a global CPR state, and CPR stage of migration will work the same way as normal migration?
> 
> I proposed a single migration stream containing pre-create state that was read early,
> but that was rejected as too complex.
> 
> I also proposed refactoring initialization so the monitor and migration streams
> could be opened earlier, but again rejected as too complex and/or not consistent with
> a long term vision for reworking initialization.
> 
>> Still2, if we pass some state in CPR it should be a kind of constant. We need a guarantee that it will not change between migration start and source stop.
>>
>>>
>>>>> So, I decided to go another way, and just migrate everything backend-related including fds through main migration channel. Of course, this requires deep reworking of device initialization in case of incoming migration (but for vhost-user-blk we need it anyway). The feature is in my series "[PATCH 00/33] vhost-user-blk: live-backend local migration" (you are in CC).
>>>
>>> You did a lot of work in those series!
>>> I suspect much less rework of initialization is required if you pass variables in cpr state.
>>
>> Not sure. I had to rework initialization anyway, as initialization damaged the connection. And this lead me to idea "if rework anyway, why not to go with one migration channel".
>>
>>>
>>>>> The success with vhost-user-blk (of-course) make me rethink TAP migration too: try to avoid using additional cpr channel and unusual waiting for QMP interface on target. And, I've just sent an RFC: "[RFC 0/7] virtio-net: live-TAP local migration"
>>>
>>> Is there a use case for this outside of CPR?
>>
>> It just works without CPR.. Will CPR bring more benefit if I enable it in the setup with my local-tap + local-vhost-user-blk capabilities ( + ignore-shared of-course)?
>>
>>
>>> CPR is intended to be the "local migration" solution that does it all :)
>>> But if you do proceed with your local migration tap solution, I would want
>>> to see that CPR could also use your code paths.
>>>
>> CPR can transparently use my code: you may enable both CPR and local-tap capability and it should work. Some devices will migrate their fds through CPR, TAP fds amd state will migrate through main migration channel. 
> 
> OK, I believe that.
> 
> I also care about cpr-exec mode.  We use it internally, and I am trying to push
> it upstream:
>    https://lore.kernel.org/qemu-devel/1755191843-283480-8-git-send-email-steven.sistare@oracle.com/
> I believe it would work with your code.  Migrated fd's in both the cpr channel and
> the main migration channel would be handled differently as shown in vmstate-types.c
> get_fd() and put_fd().  The fd is kept open across execv(), and vmstate represents
> the fd by its value (eg a small integer), rather than as an object in the unix channel.

I'm close to publish new version, which will include

> 
>> Making both channels to be unix-sockets should not be a considerable overhead I think.
>>
>> Why I like my solution more:
>>
>> - no additional channel
>> - no additional logic in management software (to handle target start with no QMP access until "migrate" command on source)
>> - less code to backport (that's personal, of course not an argument for final upstream solution)
>>
>> It seems that CPR is simpler to support as we don't need to do deep rework of initialization code.. But in reality, there is a lot of work anyway: TAP, vhost-user-blk cases proves this. You series about vfio are also huge.
> 
> TAP is the only case where we can compare both approaches, and the numbers tell
> the story:
> 
>    TAP initialization refactoring: 277 insertions(+), 308 deletions(-)

Actually, I've done a lot more refactoring than required for TAP local migration, trying to make the whole initialization more clear and consistent. And it's a good base for any modification of TAP device I think.

>    live-TAP local migration:       681 insertions(+), 72 deletions(-)

But 369 is last patch which is not for commit, and 65 a first patch with tracepoints (look at it tap_dump_packet() - thanks to AI, really helps to debug network problems, when you see packet dumps in QEMU log)
So, more honest estimate is ~250, which is in good accordance with Live update tap.

>                          total:    958 insertions(+), 380 deletions(-)
> 
>    Live update tap and vhost:      223 insertions(+), 55 deletions(-)
> 
> For any given system, if the maintainers accept the larger amount of change,
> then local migration is cool (and CPR made it possible by adding fd support
> to vmstate+QEMUFile)

Yes, native support for fds in migration API opens the doors:)

>.  But the amount of change is a harder sell.

Yes, that's right. But live-TAP isn't really big. Unlike live-vhost-user-blk unfortunately.

>> What is the benefit of CPR against simple (unix-socket) migration?

> CPR supports vfio, iommufd, and pinned memory.  Memory backend objects are
> created early, before the main migration stream is read, and squashing
> CPR into migration for those cases would require a major change in how
> qemu creates objects during live migration.

Yes, understand: less things to change in initialization code = we can cover more things..

For my downstream I need TAP, vhost-user-blk and vfio. So vfio would be the most interesting challenge, if I try to make a kind of live-vfio local migration.

- it already supported by CPR, so it would be really hard to cell 1-2 thousands of additional code lines) But I'll see, may be it will not be so much.
- we already have support in downstream, which we've never tried to send. It based on getting fds from source and passing them to target by management software.. But of course one day we should sync with upstream.

> 
> Hence CPR is the method that works for all types of objects.  The mgmt
> layer does not need to support multiple methods of live update, depending
> on what devices the VM contains.
> 

So, I'll keep my live-migrations not to be a separate mode. And they may be used as part of CPR, and I'm saving a chance to switch to CPR if needed.

Thanks for detailed answer!

-- 
Best regards,
Vladimir

Re: [RFC V2 0/8] Live update: tap and vhost

[Peter Xu]

On Tue, Sep 02, 2025 at 08:09:44PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> On 02.09.25 18:33, Steven Sistare wrote:
> > On 9/1/2025 7:44 AM, Vladimir Sementsov-Ogievskiy wrote:
> > > On 29.08.25 22:37, Steven Sistare wrote:
> > > > On 8/28/2025 11:48 AM, Steven Sistare wrote:
> > > > > On 8/23/2025 5:53 PM, Vladimir Sementsov-Ogievskiy wrote:
> > > > > > On 17.07.25 21:39, Steve Sistare wrote:
> > > > > > > Tap and vhost devices can be preserved during cpr-transfer using
> > > > > > > traditional live migration methods, wherein the management layer
> > > > > > > creates new interfaces for the target and fiddles with 'ip link'
> > > > > > > to deactivate the old interface and activate the new.
> > > > > > > 
> > > > > > > However, CPR can simply send the file descriptors to new QEMU,
> > > > > > > with no special management actions required.  The user enables
> > > > > > > this behavior by specifing '-netdev tap,cpr=on'.  The default
> > > > > > > is cpr=off.
> > > > > > 
> > > > > > Hi Steve!
> > > > > > 
> > > > > > First, me trying to test the series:
> > > > > 
> > > > > Thank-you Vladimir for all the work you are doing in this area.  I have
> > > > > reproduced the "virtio_net_set_queue_pairs: Assertion `!r' failed." bug.
> > > > > Let me dig into that before I study the larger questions you pose
> > > > > about preserving tap/vhost-user-blk in local migration versus cpr.
> > > > 
> > > > I have reproduced your journey!  I fixed the assertion, the vnet_hdr, and
> > > > the blocking fd problems which you allude to.  The attached patch fixes
> > > > them, and will be squashed into the series.
> > > > 
> > > > Ben, you also reported the !r assertion failure, so this fix should help
> > > > you also.
> > > > 
> > > > > > SOURCE:
> > > > > > 
> > > > > > sudo build/qemu-system-x86_64 -display none -vga none -device pxb-pcie,bus_nr=128,bus=pcie.0,id=pcie.1 -device pcie-root-port,id=s0,slot=0,bus=pcie.1 -device pcie-root-port,id=s1,slot=1,bus=pcie.1 -device pcie-root-port,id=s2,slot=2,bus=pcie.1 -hda /home/vsementsov/work/vms/newfocal.raw -m 4G -enable-kvm -M q35 -vnc :0 -nodefaults -vga std -qmp stdio -msg timestamp -S -object memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on -machine memory-backend=ram0 -machine aux-ram-share=on
> > > > > > 
> > > > > > {"execute": "qmp_capabilities"}
> > > > > > {"return": {}}
> > > > > > {"execute": "netdev_add", "arguments": {"cpr": true, "script": "no", "downscript": "no", "vhostforce": false, "vhost": false, "queues": 4, "ifname": "tap0", "type": "tap", "id": "netdev.1"}}
> > > > > > {"return": {}}
> > > > > > {"execute": "device_add", "arguments": {"disable-legacy": "off", "bus": "s1", "netdev": "netdev.1", "driver": "virtio-net-pci", "vectors": 18, "mq": true, "romfile": "", "mac": "d6:0d:75:f8:0f:b7", "id": "vnet.1"}}
> > > > > > {"return": {}}
> > > > > > {"execute": "cont"}
> > > > > > {"timestamp": {"seconds": 1755977653, "microseconds": 248749}, "event": "RESUME"}
> > > > > > {"return": {}}
> > > > > > {"timestamp": {"seconds": 1755977657, "microseconds": 366274}, "event": "NIC_RX_FILTER_CHANGED", "data": {"name": "vnet.1", "path": "/machine/peripheral/vnet.1/virtio-backend"}}
> > > > > > {"execute": "migrate-set-parameters", "arguments": {"mode": "cpr-transfer"}}
> > > > > > {"return": {}}
> > > > > > {"execute": "migrate", "arguments": {"channels": [{"channel-type": "main", "addr": {"path": "/tmp/migr.sock", "transport": "socket", "type": "unix"}}, {"channel-type": "cpr", "addr": {"path": "/tmp/cpr.sock", "transport": "socket", "type": "unix"}}]}}
> > > > > > {"timestamp": {"seconds": 1755977767, "microseconds": 835571}, "event": "STOP"}
> > > > > > {"return": {}}
> > > > > > 
> > > > > > TARGET:
> > > > > > 
> > > > > > sudo build/qemu-system-x86_64 -display none -vga none -device pxb-pcie,bus_nr=128,bus=pcie.0,id=pcie.1 -device pcie-root-port,id=s0,slot=0,bus=pcie.1 -device pcie-root-port,id=s1,slot=1,bus=pcie.1 -device pcie-root-port,id=s2,slot=2,bus=pcie.1 -hda /home/vsementsov/work/vms/newfocal.raw -m 4G -enable-kvm -M q35 -vnc :1 -nodefaults -vga std -qmp stdio -S -object memory-backend-file,id=ram0,size=4G,mem-p
> > > > > > ath=/dev/shm/ram0,share=on -machine memory-backend=ram0 -machine aux-ram-share=on -incoming defer -incoming '{"channel-type": "cpr","addr": { "transport": "socket","type": "unix", "path": "/tmp/cpr.sock"}}'
> > > > > > 
> > > > > > <need to wait until "migrate" on source>
> > > > > > 
> > > > > > {"execute": "qmp_capabilities"}
> > > > > > {"return": {}}
> > > > > > {"execute": "netdev_add", "arguments": {"cpr": true, "script": "no", "downscript": "no", "vhostforce": false, "vhost": false, "queues": 4, "ifname": "tap0", "type": "tap", "id": "netdev.1"}}
> > > > > > {"return": {}}
> > > > > > {"execute": "device_add", "arguments": {"disable-legacy": "off", "bus": "s1", "netdev": "netdev.1", "driver": "virtio-net-pci", "vectors": 18, "mq": true, "romfile": "", "mac": "d6:0d:75:f8:0f:b7", "id": "vnet.1"}}
> > > > > > could not disable queue
> > > > > > qemu-system-x86_64: ../hw/net/virtio-net.c:771: virtio_net_set_queue_pairs: Assertion `!r' failed.
> > > > > > fish: Job 1, 'sudo build/qemu-system-x86_64 -…' terminated by signal SIGABRT (Abort)
> > > > > > 
> > > > > > So, it crashes on device_add..
> > > > > > 
> > > > > > Second, I've come a long way, backporting you TAP v1 series together with needed parts of CPR and migration channels to QEMU 7.2, fixing different issues (like, avoid reinitialization of vnet_hdr length on target, avoid simultaneous use of tap on source an target, avoid making the fd blocking again on target), and it finally started to work.
> > > > > > 
> > > > > > But next, I went to support similar migration for vhost-user-blk, and that was a lot more complex. No reason to pass an fd in preliminary stage, when source is running (like in CPR), because:
> > > > > > 
> > > > > > 1. we just can't use the fd on target at all, until we stop use it on source, otherwise we just break vhost-user-blk protocol on the wire (unlike TAP, where some ioctls called on target doesn't break source)
> > > > > > 2. we have to pass enough additional variables, which are simpler to pass through normal migration channel (how to pass anything except fds through cpr channel?)
> > > > 
> > > > You can pass extra state through the cpr channel.  See for example vmstate_cpr_vfio_device,
> > > > and how vmstate_cpr_vfio_devices is defined as a sub-section of vmstate_cpr_state.
> > > 
> > > O, I missed this.
> > > 
> > > Hmm. Still, finally CPR becomes just an additional stage of migration, which is done prior device initialization on target..
> > > 
> > > Didn't you think of integrating it to the common scheme: so that devices may have .vmsd_cpr in addition to .vmsd? This way we don't need a global CPR state, and CPR stage of migration will work the same way as normal migration?
> > 
> > I proposed a single migration stream containing pre-create state that was read early,
> > but that was rejected as too complex.
> > 
> > I also proposed refactoring initialization so the monitor and migration streams
> > could be opened earlier, but again rejected as too complex and/or not consistent with
> > a long term vision for reworking initialization.
> > 
> > > Still2, if we pass some state in CPR it should be a kind of constant. We need a guarantee that it will not change between migration start and source stop.
> > > 
> > > > 
> > > > > > So, I decided to go another way, and just migrate everything backend-related including fds through main migration channel. Of course, this requires deep reworking of device initialization in case of incoming migration (but for vhost-user-blk we need it anyway). The feature is in my series "[PATCH 00/33] vhost-user-blk: live-backend local migration" (you are in CC).
> > > > 
> > > > You did a lot of work in those series!
> > > > I suspect much less rework of initialization is required if you pass variables in cpr state.
> > > 
> > > Not sure. I had to rework initialization anyway, as initialization damaged the connection. And this lead me to idea "if rework anyway, why not to go with one migration channel".
> > > 
> > > > 
> > > > > > The success with vhost-user-blk (of-course) make me rethink TAP migration too: try to avoid using additional cpr channel and unusual waiting for QMP interface on target. And, I've just sent an RFC: "[RFC 0/7] virtio-net: live-TAP local migration"
> > > > 
> > > > Is there a use case for this outside of CPR?
> > > 
> > > It just works without CPR.. Will CPR bring more benefit if I enable it in the setup with my local-tap + local-vhost-user-blk capabilities ( + ignore-shared of-course)?
> > > 
> > > 
> > > > CPR is intended to be the "local migration" solution that does it all :)
> > > > But if you do proceed with your local migration tap solution, I would want
> > > > to see that CPR could also use your code paths.
> > > > 
> > > CPR can transparently use my code: you may enable both CPR and
> > > local-tap capability and it should work. Some devices will migrate
> > > their fds through CPR, TAP fds amd state will migrate through main
> > > migration channel.
> > 
> > OK, I believe that.
> > 
> > I also care about cpr-exec mode.  We use it internally, and I am trying to push
> > it upstream:
> >    https://lore.kernel.org/qemu-devel/1755191843-283480-8-git-send-email-steven.sistare@oracle.com/
> > I believe it would work with your code.  Migrated fd's in both the cpr channel and
> > the main migration channel would be handled differently as shown in vmstate-types.c
> > get_fd() and put_fd().  The fd is kept open across execv(), and vmstate represents
> > the fd by its value (eg a small integer), rather than as an object in the unix channel.
> 
> I'm close to publish new version, which will include
> 
> > 
> > > Making both channels to be unix-sockets should not be a considerable overhead I think.
> > > 
> > > Why I like my solution more:
> > > 
> > > - no additional channel
> > > - no additional logic in management software (to handle target start with no QMP access until "migrate" command on source)
> > > - less code to backport (that's personal, of course not an argument for final upstream solution)
> > > 
> > > It seems that CPR is simpler to support as we don't need to do deep rework of initialization code.. But in reality, there is a lot of work anyway: TAP, vhost-user-blk cases proves this. You series about vfio are also huge.
> > 
> > TAP is the only case where we can compare both approaches, and the numbers tell
> > the story:
> > 
> >    TAP initialization refactoring: 277 insertions(+), 308 deletions(-)
> 
> Actually, I've done a lot more refactoring than required for TAP local migration, trying to make the whole initialization more clear and consistent. And it's a good base for any modification of TAP device I think.
> 
> >    live-TAP local migration:       681 insertions(+), 72 deletions(-)
> 
> But 369 is last patch which is not for commit, and 65 a first patch with tracepoints (look at it tap_dump_packet() - thanks to AI, really helps to debug network problems, when you see packet dumps in QEMU log)
> So, more honest estimate is ~250, which is in good accordance with Live update tap.
> 
> >                          total:    958 insertions(+), 380 deletions(-)
> > 
> >    Live update tap and vhost:      223 insertions(+), 55 deletions(-)
> > 
> > For any given system, if the maintainers accept the larger amount of change,
> > then local migration is cool (and CPR made it possible by adding fd support
> > to vmstate+QEMUFile)
> 
> Yes, native support for fds in migration API opens the doors:)
> 
> > .  But the amount of change is a harder sell.
> 
> Yes, that's right. But live-TAP isn't really big. Unlike live-vhost-user-blk unfortunately.
> 
> > > What is the benefit of CPR against simple (unix-socket) migration?
> 
> > CPR supports vfio, iommufd, and pinned memory.  Memory backend objects are
> > created early, before the main migration stream is read, and squashing
> > CPR into migration for those cases would require a major change in how
> > qemu creates objects during live migration.
> 
> Yes, understand: less things to change in initialization code = we can cover more things..
> 
> For my downstream I need TAP, vhost-user-blk and vfio. So vfio would be the most interesting challenge, if I try to make a kind of live-vfio local migration.
> 
> - it already supported by CPR, so it would be really hard to cell 1-2 thousands of additional code lines) But I'll see, may be it will not be so much.
> - we already have support in downstream, which we've never tried to send. It based on getting fds from source and passing them to target by management software.. But of course one day we should sync with upstream.

Sorry to jump in as late.  Just want to say that using LOCs to compare
solutions is not fair above, IMHO: we could have hacks that is a single
liner, but maintaining those can be nightmare.

PS: totally not saying that CPR is hackish! :-)

I didn't read any new code at all, I apologize if I would say stupid
things, but.. if we have cleaner way to do all of these, and if that can
happen in one channel that sounds ideal.

IIUC then we can save the is_cpr_incoming() checks all over the places -
frankly, that's part of pure hack.  It's extremely hard to maintain
longterm, IMO.

I wished devices could opt-in to provide its own model so that it is
prepared to boot the QEMU without FDs being there and pause itself at that
stage if a load would happen.  If all such is possible for all device
emulations that we would care, it'll be perfect, IMHO.  More LOCs would
deserve such refactoring (and if there're even more benefits besides
migration, which I don't know about device code but I feel so).

So I wished more of Vladimir's work land, if my understanding is correct,
and if that can competely replace the early channel some day (when every
device FDs will be able to be migrated via main channel - is it possible)?

-- 
Peter Xu

Re: [RFC V2 0/8] Live update: tap and vhost

[Chaney, Ben]

 steven.sistare@oracle.comFrom: Steve Sistare <steven.sistare@oracle.com <mailto:steven.sistare@oracle.com>

>Tap and vhost devices can be preserved during cpr-transfer using
> traditional live migration methods, wherein the management layer
> creates new interfaces for the target and fiddles with 'ip link'
> to deactivate the old interface and activate the new.

> However, CPR can simply send the file descriptors to new QEMU,
> with no special management actions required. The user enables
> this behavior by specifing '-netdev tap,cpr=on'. The default
> is cpr=off.


Hi Steve,

Thank you for sending this patch set I tried testing it, and
the migration fails with the following error on the destination:


2025-08-07T18:14:30.564323Z qemu-system-x86_64: could not disable queue
qemu-system-x86_64: ../hw/net/virtio-net.c:767: virtio_net_set_queue_pairs: Assertion `!r' failed.


And the following error on the source:

vhost_reset_device failed: Operation not permitted (1)
vhost_reset_device failed: Operation not permitted (1)
2025-08-15T14:50:16.028494Z qemu-system-x86_64: Failed to connect to 'main.sock': Connection refused
2025-08-15T14:50:16.028552Z qemu-system-x86_64: vhost_set_owner failed: Device or resource busy (16)
2025-08-15T14:50:16.028565Z qemu-system-x86_64: vhost_set_owner failed: Device or resource busy (16)
2025-08-15T14:50:16.028578Z qemu-system-x86_64: vhost_set_owner failed: Device or resource busy (16)
2025-08-15T14:50:16.028590Z qemu-system-x86_64: vhost_set_owner failed: Device or resource busy (16)
2025-08-15T14:50:16.028604Z qemu-system-x86_64: vhost_set_owner failed: Device or resource busy (16)
2025-08-15T14:50:16.028629Z qemu-system-x86_64: vhost_set_owner failed: Device or resource busy (16)
2025-08-15T14:50:16.028641Z qemu-system-x86_64: vhost_set_owner failed: Device or resource busy (16)
2025-08-15T14:50:16.028844Z qemu-system-x86_64: vhost_set_owner failed: Device or resource busy (16)
2025-08-15T14:50:16.028856Z qemu-system-x86_64: vhost_set_owner failed: Device or resource busy (16)
2025-08-15T14:50:16.028868Z qemu-system-x86_64: vhost_set_owner failed: Device or resource busy (16)
2025-08-15T14:50:16.028880Z qemu-system-x86_64: vhost_set_owner failed: Device or resource busy (16)
2025-08-15T14:50:16.028893Z qemu-system-x86_64: vhost_set_owner failed: Device or resource busy (16)
2025-08-15T14:50:16.028904Z qemu-system-x86_64: vhost_set_owner failed: Device or resource busy (16)
2025-08-15T14:50:16.028916Z qemu-system-x86_64: vhost_set_owner failed: Device or resource busy (16)
2025-08-15T14:50:16.028928Z qemu-system-x86_64: vhost_set_owner failed: Device or resource busy (16)

I suspect the issue may be related to the fact that we are dropping
privileges (-run-with user=$USERNAME) as cpr transfer has run
into other issues with that in the past, but I haven't found anything
concrete there yet.

Some other information:

The full qemu arguments used for networking are:

-netdev tap,id=net0,ifname=tap.79874411_0,script=no,downscript=no,vhost=on,queues=8,cpr=on
-device virtio-net-pci,netdev=net0,id=netpci0,mac=$mac1,vectors=18,mq=on
-netdev tap,id=net1,ifname=tap.79874411_1,script=no,downscript=no,vhost=on,queues=8,cpr=on
-device virtio-net-pci,netdev=net1,id=netpci1,mac=$mac2,vectors=18,mq=on

I applied your patch on top of 7136352b40631b058dd0fe731a0d404e761e799f
I also applied the pending arm interrupt fix

Thanks,
        Ben

Re: [RFC V2 0/8] Live update: tap and vhost

[Steven Sistare]

On 8/18/2025 11:04 AM, Chaney, Ben wrote:
>   steven.sistare@oracle.comFrom: Steve Sistare <steven.sistare@oracle.com <mailto:steven.sistare@oracle.com>
> 
>> Tap and vhost devices can be preserved during cpr-transfer using
>> traditional live migration methods, wherein the management layer
>> creates new interfaces for the target and fiddles with 'ip link'
>> to deactivate the old interface and activate the new.
> 
>> However, CPR can simply send the file descriptors to new QEMU,
>> with no special management actions required. The user enables
>> this behavior by specifing '-netdev tap,cpr=on'. The default
>> is cpr=off.
> 
> 
> Hi Steve,
> 
> Thank you for sending this patch set I tried testing it, and
> the migration fails with the following error on the destination:
> 
> 
> 2025-08-07T18:14:30.564323Z qemu-system-x86_64: could not disable queue
> qemu-system-x86_64: ../hw/net/virtio-net.c:767: virtio_net_set_queue_pairs: Assertion `!r' failed.
> 
> 
> And the following error on the source:
> 
> vhost_reset_device failed: Operation not permitted (1)
> vhost_reset_device failed: Operation not permitted (1)
> 2025-08-15T14:50:16.028494Z qemu-system-x86_64: Failed to connect to 'main.sock': Connection refused
> 2025-08-15T14:50:16.028552Z qemu-system-x86_64: vhost_set_owner failed: Device or resource busy (16)
> 2025-08-15T14:50:16.028565Z qemu-system-x86_64: vhost_set_owner failed: Device or resource busy (16)
> 2025-08-15T14:50:16.028578Z qemu-system-x86_64: vhost_set_owner failed: Device or resource busy (16)
> 2025-08-15T14:50:16.028590Z qemu-system-x86_64: vhost_set_owner failed: Device or resource busy (16)
> 2025-08-15T14:50:16.028604Z qemu-system-x86_64: vhost_set_owner failed: Device or resource busy (16)
> 2025-08-15T14:50:16.028629Z qemu-system-x86_64: vhost_set_owner failed: Device or resource busy (16)
> 2025-08-15T14:50:16.028641Z qemu-system-x86_64: vhost_set_owner failed: Device or resource busy (16)
> 2025-08-15T14:50:16.028844Z qemu-system-x86_64: vhost_set_owner failed: Device or resource busy (16)
> 2025-08-15T14:50:16.028856Z qemu-system-x86_64: vhost_set_owner failed: Device or resource busy (16)
> 2025-08-15T14:50:16.028868Z qemu-system-x86_64: vhost_set_owner failed: Device or resource busy (16)
> 2025-08-15T14:50:16.028880Z qemu-system-x86_64: vhost_set_owner failed: Device or resource busy (16)
> 2025-08-15T14:50:16.028893Z qemu-system-x86_64: vhost_set_owner failed: Device or resource busy (16)
> 2025-08-15T14:50:16.028904Z qemu-system-x86_64: vhost_set_owner failed: Device or resource busy (16)
> 2025-08-15T14:50:16.028916Z qemu-system-x86_64: vhost_set_owner failed: Device or resource busy (16)
> 2025-08-15T14:50:16.028928Z qemu-system-x86_64: vhost_set_owner failed: Device or resource busy (16)
> 
> I suspect the issue may be related to the fact that we are dropping
> privileges (-run-with user=$USERNAME) as cpr transfer has run
> into other issues with that in the past, but I haven't found anything
> concrete there yet.
> 
> Some other information:
> 
> The full qemu arguments used for networking are:
> 
> -netdev tap,id=net0,ifname=tap.79874411_0,script=no,downscript=no,vhost=on,queues=8,cpr=on
> -device virtio-net-pci,netdev=net0,id=netpci0,mac=$mac1,vectors=18,mq=on
> -netdev tap,id=net1,ifname=tap.79874411_1,script=no,downscript=no,vhost=on,queues=8,cpr=on
> -device virtio-net-pci,netdev=net1,id=netpci1,mac=$mac2,vectors=18,mq=on
> 
> I applied your patch on top of 7136352b40631b058dd0fe731a0d404e761e799f
> I also applied the pending arm interrupt fix

Thanks very much Ben!  CPR for vhost is completely broken.  I did not notice
in my testing because the "falling back on userspace virtio" code maintains
connectivity to the guest.

I will reply to "[RFC V2 5/8] Revert "vhost-backend: remove vhost_kernel_reset_device()"
and post a replacement patch which fixes the set/reset owner failures, then the remaining
calls should work.

To use run-with, you will also need the patch that you posted for setting
the owner in unix_listen_saddr, before change_process_uid is called:
   https://lore.kernel.org/qemu-devel/3D32B62F-29E2-4470-86A5-9A2B3B29E371@akamai.com/

- Steve