[PATCH v5 0/6] vhost-user-blk: add compatibility with older qemu versions

[PATCH v5 0/6] vhost-user-blk: add compatibility with older qemu versions

[Alexandr Moshkov]

v4 -> v5:
- introduce protocol feature VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN
  to guard the new message, instead of reusing the existing
  VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INLIFGHT. Update docs.
- improved commit messages.

v3 -> v4:
- add new protocol message GET_VRING_BASE_SKIP_DRAIN instead of parameter to GET_VRING_BASE message. This new message can be send instead of GET_VRING_BASE, allowing back-end to suspend inflight I/O immediately instead of waiting and completing them.
- rebase to newer master

v2 -> v3:
- fix complile problems
- add assert check in do_vhost_virtio_stop
- make inflight-migration property mutable

v1 -> v2:
- reorganize commits: make refactor commits first, then core semantic change
- add additional pre_save check for inflight migration possibility

---

This is v5 of the series previously sent as
"[PATCH v4 0/6] vhost-user-blk: fix inflight migration compatibility".

This series extends the vhost-user-blk inflight migration feature
introduced in QEMU 11.0 to address a runtime compatibility problem.

Currently, the inflight migration behaviour in vhost-user-blk is
controlled by VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT, which is
negotiated once at connection time. Once the feature is negotiated, the
back-end always uses suspend semantics on GET_VRING_BASE — there is no
way for the front-end to request normal drain behaviour on a per-call
basis without reconnecting. This makes it impossible to disable
inflight-migration at runtime, which is needed when, for example,
migrating a VM to an older QEMU version that does not support the
feature.

To solve this, the series introduces a new protocol message
VHOST_USER_GET_VRING_BASE_SKIP_DRAIN (id=45) guarded by a new protocol
feature VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN. The message is
semantically identical to GET_VRING_BASE but explicitly instructs the
back-end to suspend in-flight I/O immediately rather than draining it.
This gives the front-end explicit per-call control: GET_VRING_BASE for
normal operation, GET_VRING_BASE_SKIP_DRAIN during live migration.

In vhost-user-blk, GET_VRING_BASE_SKIP_DRAIN is sent only when
inflight-migration is enabled and the device is stopping due to live
migration (RUN_STATE_FINISH_MIGRATE). The inflight-migration property
is also made mutable so it can be toggled via qom-set at runtime without
reconnecting to the back-end.

Alexandr Moshkov (6):
  vhost-user: add skip_drain param to do_vhost_virtqueue_stop
  vhost-user: add GET_VRING_BASE_SKIP_DRAIN message
  vhost-user: use skip_drain with GET_VRING_BASE_SKIP_DRAIN message
  vhost-user-blk: make inflight-migration prop mutable
  vhost-user-blk: move inflight_needed higher
  vhost-user-blk: use GET_VRING_BASE_SKIP_DRAIN during inflight
    migration

 backends/cryptodev-vhost.c        |  2 +-
 backends/vhost-user.c             |  2 +-
 docs/interop/vhost-user.rst       | 88 +++++++++++++++++++------------
 hw/block/vhost-user-blk.c         | 43 ++++++++++++---
 hw/net/vhost_net.c                |  9 ++--
 hw/scsi/vhost-scsi-common.c       |  2 +-
 hw/virtio/vdpa-dev.c              |  2 +-
 hw/virtio/vhost-user-base.c       |  2 +-
 hw/virtio/vhost-user-fs.c         |  2 +-
 hw/virtio/vhost-user-scmi.c       |  2 +-
 hw/virtio/vhost-user.c            | 47 ++++++++++++++---
 hw/virtio/vhost-vsock-common.c    |  2 +-
 hw/virtio/vhost.c                 | 34 ++++++++----
 include/hw/virtio/vhost-backend.h |  1 +
 include/hw/virtio/vhost-user.h    |  2 +-
 include/hw/virtio/vhost.h         |  7 ++-
 16 files changed, 172 insertions(+), 75 deletions(-)

-- 
2.34.1

[PATCH v5 1/6] vhost-user: add skip_drain param to do_vhost_virtqueue_stop

[Alexandr Moshkov]

Currently do_vhost_virtqueue_stop always sends GET_VRING_BASE to the
back-end, which requires the back-end to drain all in-flight I/O before
stopping the vring.

Add a skip_drain parameter to do_vhost_virtqueue_stop and propagate it
up through vhost_virtqueue_stop, vhost_dev_stop and their callers.
The parameter will be used in a follow-up commit to send a new protocol
message that instructs the back-end to suspend in-flight I/O immediately
instead of draining it.

Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
---
 backends/cryptodev-vhost.c     |  2 +-
 backends/vhost-user.c          |  2 +-
 hw/block/vhost-user-blk.c      |  2 +-
 hw/net/vhost_net.c             |  9 +++++----
 hw/scsi/vhost-scsi-common.c    |  2 +-
 hw/virtio/vdpa-dev.c           |  2 +-
 hw/virtio/vhost-user-base.c    |  2 +-
 hw/virtio/vhost-user-fs.c      |  2 +-
 hw/virtio/vhost-user-scmi.c    |  2 +-
 hw/virtio/vhost-vsock-common.c |  2 +-
 hw/virtio/vhost.c              | 28 +++++++++++++++++++---------
 include/hw/virtio/vhost.h      |  7 +++++--
 12 files changed, 38 insertions(+), 24 deletions(-)

diff --git a/backends/cryptodev-vhost.c b/backends/cryptodev-vhost.c
index c6069f4e5b..f1ca6bcd4e 100644
--- a/backends/cryptodev-vhost.c
+++ b/backends/cryptodev-vhost.c
@@ -109,7 +109,7 @@ static void
 cryptodev_vhost_stop_one(CryptoDevBackendVhost *crypto,
                                  VirtIODevice *dev)
 {
-    vhost_dev_stop(&crypto->dev, dev, false);
+    vhost_dev_stop(&crypto->dev, dev, false, false);
     vhost_dev_disable_notifiers(&crypto->dev, dev);
 }
 
diff --git a/backends/vhost-user.c b/backends/vhost-user.c
index 380d825023..0423c2d4ec 100644
--- a/backends/vhost-user.c
+++ b/backends/vhost-user.c
@@ -108,7 +108,7 @@ vhost_user_backend_stop(VhostUserBackend *b)
         return 0;
     }
 
-    ret = vhost_dev_stop(&b->dev, b->vdev, true);
+    ret = vhost_dev_stop(&b->dev, b->vdev, true, false);
 
     if (k->set_guest_notifiers &&
         (err = k->set_guest_notifiers(qbus->parent, b->dev.nvqs, false)) < 0) {
diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
index b2e46eb3f9..e3b873af7c 100644
--- a/hw/block/vhost-user-blk.c
+++ b/hw/block/vhost-user-blk.c
@@ -232,7 +232,7 @@ static int vhost_user_blk_stop(VirtIODevice *vdev)
                  qemu_force_shutdown_requested();
 
     ret = force_stop ? vhost_dev_force_stop(&s->dev, vdev, true) :
-                       vhost_dev_stop(&s->dev, vdev, true);
+                       vhost_dev_stop(&s->dev, vdev, true, false);
 
     err = k->set_guest_notifiers(qbus->parent, s->dev.nvqs, false);
     if (err < 0) {
diff --git a/hw/net/vhost_net.c b/hw/net/vhost_net.c
index 323d117735..6e05c995f1 100644
--- a/hw/net/vhost_net.c
+++ b/hw/net/vhost_net.c
@@ -384,7 +384,7 @@ fail:
     if (net->nc->info->poll) {
         net->nc->info->poll(net->nc, true);
     }
-    vhost_dev_stop(&net->dev, dev, false);
+    vhost_dev_stop(&net->dev, dev, false, false);
 fail_start:
     return r;
 }
@@ -403,7 +403,7 @@ static void vhost_net_stop_one(struct vhost_net *net,
     if (net->nc->info->poll) {
         net->nc->info->poll(net->nc, true);
     }
-    vhost_dev_stop(&net->dev, dev, false);
+    vhost_dev_stop(&net->dev, dev, false, false);
     if (net->nc->info->stop) {
         net->nc->info->stop(net->nc);
     }
@@ -636,7 +636,8 @@ void vhost_net_virtqueue_reset(VirtIODevice *vdev, NetClientState *nc,
     vhost_virtqueue_stop(&net->dev,
                          vdev,
                          net->dev.vqs + idx,
-                         net->dev.vq_index + idx);
+                         net->dev.vq_index + idx,
+                         false);
 }
 
 int vhost_net_virtqueue_restart(VirtIODevice *vdev, NetClientState *nc,
@@ -686,7 +687,7 @@ err_start:
         assert(ret >= 0);
     }
 
-    vhost_dev_stop(&net->dev, vdev, false);
+    vhost_dev_stop(&net->dev, vdev, false, false);
 
     return r;
 }
diff --git a/hw/scsi/vhost-scsi-common.c b/hw/scsi/vhost-scsi-common.c
index e19800a0bc..e546a6dc75 100644
--- a/hw/scsi/vhost-scsi-common.c
+++ b/hw/scsi/vhost-scsi-common.c
@@ -108,7 +108,7 @@ int vhost_scsi_common_stop(VHostSCSICommon *vsc)
     VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(qbus);
     int ret = 0;
 
-    ret = vhost_dev_stop(&vsc->dev, vdev, true);
+    ret = vhost_dev_stop(&vsc->dev, vdev, true, false);
 
     if (k->set_guest_notifiers) {
         int r = k->set_guest_notifiers(qbus->parent, vsc->dev.nvqs, false);
diff --git a/hw/virtio/vdpa-dev.c b/hw/virtio/vdpa-dev.c
index 94188d37bb..b86d0b664a 100644
--- a/hw/virtio/vdpa-dev.c
+++ b/hw/virtio/vdpa-dev.c
@@ -300,7 +300,7 @@ static void vhost_vdpa_device_stop(VirtIODevice *vdev)
         return;
     }
 
-    vhost_dev_stop(&s->dev, vdev, false);
+    vhost_dev_stop(&s->dev, vdev, false, false);
 
     ret = k->set_guest_notifiers(qbus->parent, s->dev.nvqs, false);
     if (ret < 0) {
diff --git a/hw/virtio/vhost-user-base.c b/hw/virtio/vhost-user-base.c
index 39b5e637fc..abeae1a064 100644
--- a/hw/virtio/vhost-user-base.c
+++ b/hw/virtio/vhost-user-base.c
@@ -78,7 +78,7 @@ static int vub_stop(VirtIODevice *vdev)
         return 0;
     }
 
-    ret = vhost_dev_stop(&vub->vhost_dev, vdev, true);
+    ret = vhost_dev_stop(&vub->vhost_dev, vdev, true, false);
 
     err = k->set_guest_notifiers(qbus->parent, vub->vhost_dev.nvqs, false);
     if (err < 0) {
diff --git a/hw/virtio/vhost-user-fs.c b/hw/virtio/vhost-user-fs.c
index 209993918a..0d8842817a 100644
--- a/hw/virtio/vhost-user-fs.c
+++ b/hw/virtio/vhost-user-fs.c
@@ -111,7 +111,7 @@ static int vuf_stop(VirtIODevice *vdev)
         return 0;
     }
 
-    ret = vhost_dev_stop(&fs->vhost_dev, vdev, true);
+    ret = vhost_dev_stop(&fs->vhost_dev, vdev, true, false);
 
     err = k->set_guest_notifiers(qbus->parent, fs->vhost_dev.nvqs, false);
     if (err < 0) {
diff --git a/hw/virtio/vhost-user-scmi.c b/hw/virtio/vhost-user-scmi.c
index 9470f68c1f..9fa65d7d8b 100644
--- a/hw/virtio/vhost-user-scmi.c
+++ b/hw/virtio/vhost-user-scmi.c
@@ -101,7 +101,7 @@ static int vu_scmi_stop(VirtIODevice *vdev)
         return 0;
     }
 
-    ret = vhost_dev_stop(vhost_dev, vdev, true);
+    ret = vhost_dev_stop(vhost_dev, vdev, true, false);
 
     err = k->set_guest_notifiers(qbus->parent, vhost_dev->nvqs, false);
     if (err < 0) {
diff --git a/hw/virtio/vhost-vsock-common.c b/hw/virtio/vhost-vsock-common.c
index b79f4c9ce6..4ef037627b 100644
--- a/hw/virtio/vhost-vsock-common.c
+++ b/hw/virtio/vhost-vsock-common.c
@@ -106,7 +106,7 @@ int vhost_vsock_common_stop(VirtIODevice *vdev)
         return 0;
     }
 
-    ret = vhost_dev_stop(&vvc->vhost_dev, vdev, true);
+    ret = vhost_dev_stop(&vvc->vhost_dev, vdev, true, false);
 
     err = k->set_guest_notifiers(qbus->parent, vvc->vhost_dev.nvqs, false);
     if (err < 0) {
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index af41841b52..2a62550222 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -23,6 +23,7 @@
 #include "qemu/log.h"
 #include "standard-headers/linux/vhost_types.h"
 #include "hw/virtio/virtio-bus.h"
+#include "hw/virtio/vhost-user.h"
 #include "hw/mem/memory-device.h"
 #include "migration/blocker.h"
 #include "migration/qemu-file-types.h"
@@ -1496,8 +1497,13 @@ fail:
 static int do_vhost_virtqueue_stop(struct vhost_dev *dev,
                                    struct VirtIODevice *vdev,
                                    struct vhost_virtqueue *vq,
-                                   unsigned idx, bool force)
+                                   unsigned idx, bool force,
+                                   bool skip_drain)
 {
+    if (skip_drain) {
+        assert(vhost_user_has_protocol_feature(dev,
+               VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT));
+    }
     int vhost_vq_index = dev->vhost_ops->vhost_get_vq_index(dev, idx);
     struct vhost_vring_state state = {
         .index = vhost_vq_index,
@@ -1547,9 +1553,10 @@ static int do_vhost_virtqueue_stop(struct vhost_dev *dev,
 int vhost_virtqueue_stop(struct vhost_dev *dev,
                          struct VirtIODevice *vdev,
                          struct vhost_virtqueue *vq,
-                         unsigned idx)
+                         unsigned idx,
+                         bool skip_drain)
 {
-    return do_vhost_virtqueue_stop(dev, vdev, vq, idx, false);
+    return do_vhost_virtqueue_stop(dev, vdev, vq, idx, false, skip_drain);
 }
 
 static int vhost_virtqueue_set_busyloop_timeout(struct vhost_dev *dev,
@@ -2302,7 +2309,8 @@ fail_vq:
         vhost_virtqueue_stop(hdev,
                              vdev,
                              hdev->vqs + i,
-                             hdev->vq_index + i);
+                             hdev->vq_index + i,
+                             false);
     }
 
 fail_mem:
@@ -2317,7 +2325,7 @@ fail_features:
 
 /* Host notifiers must be enabled at this point. */
 static int do_vhost_dev_stop(struct vhost_dev *hdev, VirtIODevice *vdev,
-                             bool vrings, bool force)
+                             bool vrings, bool force, bool skip_drain)
 {
     int i;
     int rc = 0;
@@ -2344,7 +2352,8 @@ static int do_vhost_dev_stop(struct vhost_dev *hdev, VirtIODevice *vdev,
                                       vdev,
                                       hdev->vqs + i,
                                       hdev->vq_index + i,
-                                      force);
+                                      force,
+                                      skip_drain);
     }
     if (hdev->vhost_ops->vhost_reset_status) {
         hdev->vhost_ops->vhost_reset_status(hdev);
@@ -2366,15 +2375,16 @@ static int do_vhost_dev_stop(struct vhost_dev *hdev, VirtIODevice *vdev,
     return rc;
 }
 
-int vhost_dev_stop(struct vhost_dev *hdev, VirtIODevice *vdev, bool vrings)
+int vhost_dev_stop(struct vhost_dev *hdev, VirtIODevice *vdev, bool vrings,
+                   bool skip_drain)
 {
-    return do_vhost_dev_stop(hdev, vdev, vrings, false);
+    return do_vhost_dev_stop(hdev, vdev, vrings, false, skip_drain);
 }
 
 int vhost_dev_force_stop(struct vhost_dev *hdev, VirtIODevice *vdev,
                          bool vrings)
 {
-    return do_vhost_dev_stop(hdev, vdev, vrings, true);
+    return do_vhost_dev_stop(hdev, vdev, vrings, true, false);
 }
 
 int vhost_net_set_backend(struct vhost_dev *hdev,
diff --git a/include/hw/virtio/vhost.h b/include/hw/virtio/vhost.h
index 684bafcaad..33b88f95ea 100644
--- a/include/hw/virtio/vhost.h
+++ b/include/hw/virtio/vhost.h
@@ -228,6 +228,7 @@ int vhost_dev_start(struct vhost_dev *hdev, VirtIODevice *vdev, bool vrings);
  * @hdev: common vhost_dev structure
  * @vdev: the VirtIODevice structure
  * @vrings: true to have vrings disabled in this call
+ * @skip_drain: true to notice back-end to skip draining all in-flight requests
  *
  * Stop the vhost device. After the device is stopped the notifiers
  * can be disabled (@vhost_dev_disable_notifiers) and the device can
@@ -235,7 +236,8 @@ int vhost_dev_start(struct vhost_dev *hdev, VirtIODevice *vdev, bool vrings);
  *
  * Return: 0 on success, != 0 on error when stopping dev.
  */
-int vhost_dev_stop(struct vhost_dev *hdev, VirtIODevice *vdev, bool vrings);
+int vhost_dev_stop(struct vhost_dev *hdev, VirtIODevice *vdev, bool vrings,
+                   bool skip_drain);
 
 /**
  * vhost_dev_force_stop() - force stop the vhost device
@@ -393,7 +395,8 @@ int vhost_device_iotlb_miss(struct vhost_dev *dev, uint64_t iova, int write);
 int vhost_virtqueue_start(struct vhost_dev *dev, struct VirtIODevice *vdev,
                           struct vhost_virtqueue *vq, unsigned idx);
 int vhost_virtqueue_stop(struct vhost_dev *dev, struct VirtIODevice *vdev,
-                         struct vhost_virtqueue *vq, unsigned idx);
+                         struct vhost_virtqueue *vq, unsigned idx,
+                         bool skip_drain);
 
 void vhost_dev_reset_inflight(struct vhost_inflight *inflight);
 void vhost_dev_free_inflight(struct vhost_inflight *inflight);
-- 
2.34.1

[PATCH v5 2/6] vhost-user: add GET_VRING_BASE_SKIP_DRAIN message

[Alexandr Moshkov]

VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT was introduced to allow
the back-end to suspend in-flight I/O during GET_VRING_BASE instead of
draining it, enabling live migration of in-flight requests. However,
this behaviour is tied to the protocol feature itself — once negotiated,
there is no way for the front-end to tell the back-end to fall back to
the normal drain behaviour on a per-stop basis. This makes it impossible
to selectively use suspend semantics only during live migration while
keeping drain semantics in other cases.

Introduce a separate message VHOST_USER_GET_VRING_BASE_SKIP_DRAIN (id=45)
guarded by a new protocol feature
VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN. The message is identical
to GET_VRING_BASE except that the back-end must immediately suspend all
in-flight I/O and record it in the inflight region. This way the front-end
has explicit per-call control: send GET_VRING_BASE for normal drain, send
GET_VRING_BASE_SKIP_DRAIN when immediate suspend is needed.

The new feature requires both VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT
and VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD to be negotiated.

Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
---
 docs/interop/vhost-user.rst       | 88 +++++++++++++++++++------------
 hw/virtio/vhost-user.c            | 44 ++++++++++++++--
 hw/virtio/vhost.c                 |  2 +-
 include/hw/virtio/vhost-backend.h |  1 +
 include/hw/virtio/vhost-user.h    |  1 +
 5 files changed, 95 insertions(+), 41 deletions(-)

diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
index c83ae2accb..d2e0efaeaa 100644
--- a/docs/interop/vhost-user.rst
+++ b/docs/interop/vhost-user.rst
@@ -445,6 +445,7 @@ replies, except for the following requests:
 * ``VHOST_USER_GET_FEATURES``
 * ``VHOST_USER_GET_PROTOCOL_FEATURES``
 * ``VHOST_USER_GET_VRING_BASE``
+* ``VHOST_USER_GET_VRING_BASE_SKIP_DRAIN``
 * ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
 * ``VHOST_USER_GET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
 
@@ -523,7 +524,7 @@ must start a ring upon receiving a kick (that is, detecting that file
 descriptor is readable) on the descriptor specified by
 ``VHOST_USER_SET_VRING_KICK`` or receiving the in-band message
 ``VHOST_USER_VRING_KICK`` if negotiated, and stop a ring upon receiving
-``VHOST_USER_GET_VRING_BASE``.
+``VHOST_USER_GET_VRING_BASE`` or ``VHOST_USER_GET_VRING_BASE_SKIP_DRAIN``.
 
 Rings can be enabled or disabled by ``VHOST_USER_SET_VRING_ENABLE``.
 
@@ -1101,29 +1102,30 @@ Protocol features
 
 .. code:: c
 
-  #define VHOST_USER_PROTOCOL_F_MQ                       0
-  #define VHOST_USER_PROTOCOL_F_LOG_SHMFD                1
-  #define VHOST_USER_PROTOCOL_F_RARP                     2
-  #define VHOST_USER_PROTOCOL_F_REPLY_ACK                3
-  #define VHOST_USER_PROTOCOL_F_MTU                      4
-  #define VHOST_USER_PROTOCOL_F_BACKEND_REQ              5
-  #define VHOST_USER_PROTOCOL_F_CROSS_ENDIAN             6
-  #define VHOST_USER_PROTOCOL_F_CRYPTO_SESSION           7
-  #define VHOST_USER_PROTOCOL_F_PAGEFAULT                8
-  #define VHOST_USER_PROTOCOL_F_CONFIG                   9
-  #define VHOST_USER_PROTOCOL_F_BACKEND_SEND_FD         10
-  #define VHOST_USER_PROTOCOL_F_HOST_NOTIFIER           11
-  #define VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD          12
-  #define VHOST_USER_PROTOCOL_F_RESET_DEVICE            13
-  #define VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS    14
-  #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS     15
-  #define VHOST_USER_PROTOCOL_F_STATUS                  16
-  #define VHOST_USER_PROTOCOL_F_XEN_MMAP                17
-  #define VHOST_USER_PROTOCOL_F_SHARED_OBJECT           18
-  #define VHOST_USER_PROTOCOL_F_DEVICE_STATE            19
-  #define VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT 20
-  #define VHOST_USER_PROTOCOL_F_GPA_ADDRESSES           21
-  #define VHOST_USER_PROTOCOL_F_SHMEM_MAP               22
+  #define VHOST_USER_PROTOCOL_F_MQ                         0
+  #define VHOST_USER_PROTOCOL_F_LOG_SHMFD                  1
+  #define VHOST_USER_PROTOCOL_F_RARP                       2
+  #define VHOST_USER_PROTOCOL_F_REPLY_ACK                  3
+  #define VHOST_USER_PROTOCOL_F_MTU                        4
+  #define VHOST_USER_PROTOCOL_F_BACKEND_REQ                5
+  #define VHOST_USER_PROTOCOL_F_CROSS_ENDIAN               6
+  #define VHOST_USER_PROTOCOL_F_CRYPTO_SESSION             7
+  #define VHOST_USER_PROTOCOL_F_PAGEFAULT                  8
+  #define VHOST_USER_PROTOCOL_F_CONFIG                     9
+  #define VHOST_USER_PROTOCOL_F_BACKEND_SEND_FD           10
+  #define VHOST_USER_PROTOCOL_F_HOST_NOTIFIER             11
+  #define VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD            12
+  #define VHOST_USER_PROTOCOL_F_RESET_DEVICE              13
+  #define VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS      14
+  #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS       15
+  #define VHOST_USER_PROTOCOL_F_STATUS                    16
+  #define VHOST_USER_PROTOCOL_F_XEN_MMAP                  17
+  #define VHOST_USER_PROTOCOL_F_SHARED_OBJECT             18
+  #define VHOST_USER_PROTOCOL_F_DEVICE_STATE              19
+  #define VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT   20
+  #define VHOST_USER_PROTOCOL_F_GPA_ADDRESSES             21
+  #define VHOST_USER_PROTOCOL_F_SHMEM_MAP                 22
+  #define VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN 23
 
 Front-end message types
 -----------------------
@@ -1320,17 +1322,11 @@ Front-end message types
   set to 0.
 
   By default, the back-end must complete all inflight I/O requests for the
-  specified vring before stopping it.
-
-  If the ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT`` protocol
-  feature has been negotiated, the back-end may suspend in-flight I/O
-  requests and record them as described in :ref:`Inflight I/O tracking
-  <inflight_io_tracking>` instead of completing them before stopping the vring.
-  How to suspend an in-flight request depends on the implementation of the back-end
-  but it typically can be done by aborting or cancelling the underlying I/O
-  request. The ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT``
-  protocol feature must only be negotiated if
-  ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD`` is also negotiated.
+  specified vring before stopping it. If the
+  ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN`` protocol feature has
+  been negotiated, the front-end may instead use
+  ``VHOST_USER_GET_VRING_BASE_SKIP_DRAIN`` to request the back-end to
+  suspend in-flight I/O immediately.
 
 ``VHOST_USER_SET_VRING_KICK``
   :id: 12
@@ -1833,6 +1829,28 @@ Front-end message types
 
   * The size may be 0 if the region is unused.
 
+``VHOST_USER_GET_VRING_BASE_SKIP_DRAIN``
+  :id: 45
+  :equivalent ioctl: N/A
+  :request payload: vring state description
+  :reply payload: vring descriptor index/indices
+
+  This message requires the ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN``
+  protocol feature to be negotiated.
+
+  Identical to ``VHOST_USER_GET_VRING_BASE`` except that the back-end
+  must not wait for inflight I/O requests to complete before stopping
+  the vring.  Instead, the back-end must immediately suspend all
+  in-flight I/O requests and record them as described in
+  :ref:`Inflight I/O tracking <inflight_io_tracking>`. How to suspend
+  an in-flight request depends on the implementation of the back-end,
+  but it typically can be done by aborting or cancelling the underlying
+  I/O request.
+
+  The ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN`` protocol feature
+  must only be negotiated if both ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT``
+  and ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD`` are also negotiated.
+
 Back-end message types
 ----------------------
 
diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c
index d627351f45..bf455daa44 100644
--- a/hw/virtio/vhost-user.c
+++ b/hw/virtio/vhost-user.c
@@ -101,6 +101,7 @@ typedef enum VhostUserRequest {
     VHOST_USER_SET_DEVICE_STATE_FD = 42,
     VHOST_USER_CHECK_DEVICE_STATE = 43,
     VHOST_USER_GET_SHMEM_CONFIG = 44,
+    VHOST_USER_GET_VRING_BASE_SKIP_DRAIN = 45,
     VHOST_USER_MAX
 } VhostUserRequest;
 
@@ -167,6 +168,7 @@ static const char *vhost_req_name(VhostUserRequest req)
     VHOST_USER_CASE(GET_SHARED_OBJECT)
     VHOST_USER_CASE(SET_DEVICE_STATE_FD)
     VHOST_USER_CASE(CHECK_DEVICE_STATE)
+    VHOST_USER_CASE(GET_VRING_BASE_SKIP_DRAIN)
     default:
         return "<unknown>";
     }
@@ -1399,12 +1401,18 @@ static VhostUserHostNotifier *fetch_notifier(VhostUserState *u,
     return g_ptr_array_index(u->notifiers, idx);
 }
 
-static int vhost_user_get_vring_base(struct vhost_dev *dev,
-                                     struct vhost_vring_state *ring)
+static int get_vring_base(struct vhost_dev *dev,
+                          struct vhost_vring_state *ring,
+                          bool skip_drain)
 {
     int ret;
+    int request = VHOST_USER_GET_VRING_BASE;
+    if (skip_drain) {
+        request = VHOST_USER_GET_VRING_BASE_SKIP_DRAIN;
+    }
+
     VhostUserMsg msg = {
-        .hdr.request = VHOST_USER_GET_VRING_BASE,
+        .hdr.request = request,
         .hdr.flags = VHOST_USER_VERSION,
         .payload.state = *ring,
         .hdr.size = sizeof(msg.payload.state),
@@ -1424,9 +1432,9 @@ static int vhost_user_get_vring_base(struct vhost_dev *dev,
         return ret;
     }
 
-    if (msg.hdr.request != VHOST_USER_GET_VRING_BASE) {
+    if (msg.hdr.request != request) {
         error_report("Received unexpected msg type. Expected %d received %d",
-                     VHOST_USER_GET_VRING_BASE, msg.hdr.request);
+                     request, msg.hdr.request);
         return -EPROTO;
     }
 
@@ -1440,6 +1448,25 @@ static int vhost_user_get_vring_base(struct vhost_dev *dev,
     return 0;
 }
 
+static int vhost_user_get_vring_base(struct vhost_dev *dev,
+                                     struct vhost_vring_state *ring)
+{
+    return get_vring_base(dev, ring, false);
+}
+
+static int vhost_user_get_vring_base_skip_drain(struct vhost_dev *dev,
+                                     struct vhost_vring_state *ring)
+{
+    bool skip_drain_supported = vhost_user_has_protocol_feature(dev,
+                               VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN);
+
+    if (!skip_drain_supported) {
+        return 0;
+    }
+
+    return get_vring_base(dev, ring, true);
+}
+
 static int vhost_set_vring_file(struct vhost_dev *dev,
                                 VhostUserRequest request,
                                 struct vhost_vring_file *file)
@@ -2551,6 +2578,12 @@ static int vhost_user_backend_init(struct vhost_dev *dev, void *opaque,
                                VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
         }
 
+        if (!virtio_has_feature(protocol_features,
+                                VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT)) {
+            protocol_features &= ~(1ULL <<
+                             VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN);
+        }
+
         /* final set of protocol features */
         u->protocol_features = protocol_features;
         err = vhost_user_set_protocol_features(dev, u->protocol_features);
@@ -3409,6 +3442,7 @@ const VhostOps user_ops = {
         .vhost_set_vring_num = vhost_user_set_vring_num,
         .vhost_set_vring_base = vhost_user_set_vring_base,
         .vhost_get_vring_base = vhost_user_get_vring_base,
+        .vhost_get_vring_base_skip_drain = vhost_user_get_vring_base_skip_drain,
         .vhost_set_vring_kick = vhost_user_set_vring_kick,
         .vhost_set_vring_call = vhost_user_set_vring_call,
         .vhost_set_vring_err = vhost_user_set_vring_err,
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index 2a62550222..2c7f464d55 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -1502,7 +1502,7 @@ static int do_vhost_virtqueue_stop(struct vhost_dev *dev,
 {
     if (skip_drain) {
         assert(vhost_user_has_protocol_feature(dev,
-               VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT));
+               VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN));
     }
     int vhost_vq_index = dev->vhost_ops->vhost_get_vq_index(dev, idx);
     struct vhost_vring_state state = {
diff --git a/include/hw/virtio/vhost-backend.h b/include/hw/virtio/vhost-backend.h
index d878d7b733..daa979a1aa 100644
--- a/include/hw/virtio/vhost-backend.h
+++ b/include/hw/virtio/vhost-backend.h
@@ -190,6 +190,7 @@ typedef struct VhostOps {
     vhost_set_vring_num_op vhost_set_vring_num;
     vhost_set_vring_base_op vhost_set_vring_base;
     vhost_get_vring_base_op vhost_get_vring_base;
+    vhost_get_vring_base_op vhost_get_vring_base_skip_drain;
     vhost_set_vring_kick_op vhost_set_vring_kick;
     vhost_set_vring_call_op vhost_set_vring_call;
     vhost_set_vring_err_op vhost_set_vring_err;
diff --git a/include/hw/virtio/vhost-user.h b/include/hw/virtio/vhost-user.h
index 06c360af18..47c13f8677 100644
--- a/include/hw/virtio/vhost-user.h
+++ b/include/hw/virtio/vhost-user.h
@@ -36,6 +36,7 @@ enum VhostUserProtocolFeature {
     VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT = 20,
     VHOST_USER_PROTOCOL_F_GPA_ADDRESSES = 21,
     VHOST_USER_PROTOCOL_F_SHMEM = 22,
+    VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN = 23,
     VHOST_USER_PROTOCOL_F_MAX
 };
 
-- 
2.34.1

Re: [PATCH v5 2/6] vhost-user: add GET_VRING_BASE_SKIP_DRAIN message

[Michael S. Tsirkin]

On Tue, Jul 28, 2026 at 03:08:37PM +0500, Alexandr Moshkov wrote:
> VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT was introduced to allow
> the back-end to suspend in-flight I/O during GET_VRING_BASE instead of
> draining it, enabling live migration of in-flight requests. However,
> this behaviour is tied to the protocol feature itself — once negotiated,

Can we please avoid em-dashes and all kind of unicode fanciness in
commit log? I worry if we do this the next step is emojis) Let's stick
to ASCII with the exception of UTF-8 in contributor names.

Thanks.

> there is no way for the front-end to tell the back-end to fall back to
> the normal drain behaviour on a per-stop basis. This makes it impossible
> to selectively use suspend semantics only during live migration while
> keeping drain semantics in other cases.


So what are the "other cases"?

> 
> Introduce a separate message VHOST_USER_GET_VRING_BASE_SKIP_DRAIN (id=45)
> guarded by a new protocol feature
> VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN. The message is identical
> to GET_VRING_BASE except that the back-end must immediately suspend all
> in-flight I/O and record it in the inflight region. This way the front-end
> has explicit per-call control: send GET_VRING_BASE for normal drain, send
> GET_VRING_BASE_SKIP_DRAIN when immediate suspend is needed.
> 
> The new feature requires both VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT
> and VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD to be negotiated.
> 
> Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
> ---
>  docs/interop/vhost-user.rst       | 88 +++++++++++++++++++------------
>  hw/virtio/vhost-user.c            | 44 ++++++++++++++--
>  hw/virtio/vhost.c                 |  2 +-
>  include/hw/virtio/vhost-backend.h |  1 +
>  include/hw/virtio/vhost-user.h    |  1 +
>  5 files changed, 95 insertions(+), 41 deletions(-)
> 
> diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
> index c83ae2accb..d2e0efaeaa 100644
> --- a/docs/interop/vhost-user.rst
> +++ b/docs/interop/vhost-user.rst
> @@ -445,6 +445,7 @@ replies, except for the following requests:
>  * ``VHOST_USER_GET_FEATURES``
>  * ``VHOST_USER_GET_PROTOCOL_FEATURES``
>  * ``VHOST_USER_GET_VRING_BASE``
> +* ``VHOST_USER_GET_VRING_BASE_SKIP_DRAIN``
>  * ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
>  * ``VHOST_USER_GET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
>  
> @@ -523,7 +524,7 @@ must start a ring upon receiving a kick (that is, detecting that file
>  descriptor is readable) on the descriptor specified by
>  ``VHOST_USER_SET_VRING_KICK`` or receiving the in-band message
>  ``VHOST_USER_VRING_KICK`` if negotiated, and stop a ring upon receiving
> -``VHOST_USER_GET_VRING_BASE``.
> +``VHOST_USER_GET_VRING_BASE`` or ``VHOST_USER_GET_VRING_BASE_SKIP_DRAIN``.
>  
>  Rings can be enabled or disabled by ``VHOST_USER_SET_VRING_ENABLE``.
>  
> @@ -1101,29 +1102,30 @@ Protocol features
>  
>  .. code:: c
>  
> -  #define VHOST_USER_PROTOCOL_F_MQ                       0
> -  #define VHOST_USER_PROTOCOL_F_LOG_SHMFD                1
> -  #define VHOST_USER_PROTOCOL_F_RARP                     2
> -  #define VHOST_USER_PROTOCOL_F_REPLY_ACK                3
> -  #define VHOST_USER_PROTOCOL_F_MTU                      4
> -  #define VHOST_USER_PROTOCOL_F_BACKEND_REQ              5
> -  #define VHOST_USER_PROTOCOL_F_CROSS_ENDIAN             6
> -  #define VHOST_USER_PROTOCOL_F_CRYPTO_SESSION           7
> -  #define VHOST_USER_PROTOCOL_F_PAGEFAULT                8
> -  #define VHOST_USER_PROTOCOL_F_CONFIG                   9
> -  #define VHOST_USER_PROTOCOL_F_BACKEND_SEND_FD         10
> -  #define VHOST_USER_PROTOCOL_F_HOST_NOTIFIER           11
> -  #define VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD          12
> -  #define VHOST_USER_PROTOCOL_F_RESET_DEVICE            13
> -  #define VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS    14
> -  #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS     15
> -  #define VHOST_USER_PROTOCOL_F_STATUS                  16
> -  #define VHOST_USER_PROTOCOL_F_XEN_MMAP                17
> -  #define VHOST_USER_PROTOCOL_F_SHARED_OBJECT           18
> -  #define VHOST_USER_PROTOCOL_F_DEVICE_STATE            19
> -  #define VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT 20
> -  #define VHOST_USER_PROTOCOL_F_GPA_ADDRESSES           21
> -  #define VHOST_USER_PROTOCOL_F_SHMEM_MAP               22
> +  #define VHOST_USER_PROTOCOL_F_MQ                         0
> +  #define VHOST_USER_PROTOCOL_F_LOG_SHMFD                  1
> +  #define VHOST_USER_PROTOCOL_F_RARP                       2
> +  #define VHOST_USER_PROTOCOL_F_REPLY_ACK                  3
> +  #define VHOST_USER_PROTOCOL_F_MTU                        4
> +  #define VHOST_USER_PROTOCOL_F_BACKEND_REQ                5
> +  #define VHOST_USER_PROTOCOL_F_CROSS_ENDIAN               6
> +  #define VHOST_USER_PROTOCOL_F_CRYPTO_SESSION             7
> +  #define VHOST_USER_PROTOCOL_F_PAGEFAULT                  8
> +  #define VHOST_USER_PROTOCOL_F_CONFIG                     9
> +  #define VHOST_USER_PROTOCOL_F_BACKEND_SEND_FD           10
> +  #define VHOST_USER_PROTOCOL_F_HOST_NOTIFIER             11
> +  #define VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD            12
> +  #define VHOST_USER_PROTOCOL_F_RESET_DEVICE              13
> +  #define VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS      14
> +  #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS       15
> +  #define VHOST_USER_PROTOCOL_F_STATUS                    16
> +  #define VHOST_USER_PROTOCOL_F_XEN_MMAP                  17
> +  #define VHOST_USER_PROTOCOL_F_SHARED_OBJECT             18
> +  #define VHOST_USER_PROTOCOL_F_DEVICE_STATE              19
> +  #define VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT   20
> +  #define VHOST_USER_PROTOCOL_F_GPA_ADDRESSES             21
> +  #define VHOST_USER_PROTOCOL_F_SHMEM_MAP                 22
> +  #define VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN 23
>  
>  Front-end message types
>  -----------------------
> @@ -1320,17 +1322,11 @@ Front-end message types
>    set to 0.
>  
>    By default, the back-end must complete all inflight I/O requests for the
> -  specified vring before stopping it.
> -
> -  If the ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT`` protocol
> -  feature has been negotiated, the back-end may suspend in-flight I/O
> -  requests and record them as described in :ref:`Inflight I/O tracking
> -  <inflight_io_tracking>` instead of completing them before stopping the vring.
> -  How to suspend an in-flight request depends on the implementation of the back-end
> -  but it typically can be done by aborting or cancelling the underlying I/O
> -  request. The ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT``
> -  protocol feature must only be negotiated if
> -  ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD`` is also negotiated.
> +  specified vring before stopping it. If the
> +  ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN`` protocol feature has
> +  been negotiated, the front-end may instead use
> +  ``VHOST_USER_GET_VRING_BASE_SKIP_DRAIN`` to request the back-end to
> +  suspend in-flight I/O immediately.
>  
>  ``VHOST_USER_SET_VRING_KICK``
>    :id: 12
> @@ -1833,6 +1829,28 @@ Front-end message types
>  
>    * The size may be 0 if the region is unused.
>  
> +``VHOST_USER_GET_VRING_BASE_SKIP_DRAIN``
> +  :id: 45
> +  :equivalent ioctl: N/A
> +  :request payload: vring state description
> +  :reply payload: vring descriptor index/indices
> +
> +  This message requires the ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN``
> +  protocol feature to be negotiated.
> +
> +  Identical to ``VHOST_USER_GET_VRING_BASE`` except that the back-end
> +  must not wait for inflight I/O requests to complete before stopping
> +  the vring.  Instead, the back-end must immediately suspend all
> +  in-flight I/O requests and record them as described in
> +  :ref:`Inflight I/O tracking <inflight_io_tracking>`. How to suspend
> +  an in-flight request depends on the implementation of the back-end,
> +  but it typically can be done by aborting or cancelling the underlying
> +  I/O request.
> +
> +  The ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN`` protocol feature
> +  must only be negotiated if both ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT``
> +  and ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD`` are also negotiated.
> +
>  Back-end message types
>  ----------------------
>  
> diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c
> index d627351f45..bf455daa44 100644
> --- a/hw/virtio/vhost-user.c
> +++ b/hw/virtio/vhost-user.c
> @@ -101,6 +101,7 @@ typedef enum VhostUserRequest {
>      VHOST_USER_SET_DEVICE_STATE_FD = 42,
>      VHOST_USER_CHECK_DEVICE_STATE = 43,
>      VHOST_USER_GET_SHMEM_CONFIG = 44,
> +    VHOST_USER_GET_VRING_BASE_SKIP_DRAIN = 45,
>      VHOST_USER_MAX
>  } VhostUserRequest;
>  
> @@ -167,6 +168,7 @@ static const char *vhost_req_name(VhostUserRequest req)
>      VHOST_USER_CASE(GET_SHARED_OBJECT)
>      VHOST_USER_CASE(SET_DEVICE_STATE_FD)
>      VHOST_USER_CASE(CHECK_DEVICE_STATE)
> +    VHOST_USER_CASE(GET_VRING_BASE_SKIP_DRAIN)
>      default:
>          return "<unknown>";
>      }
> @@ -1399,12 +1401,18 @@ static VhostUserHostNotifier *fetch_notifier(VhostUserState *u,
>      return g_ptr_array_index(u->notifiers, idx);
>  }
>  
> -static int vhost_user_get_vring_base(struct vhost_dev *dev,
> -                                     struct vhost_vring_state *ring)
> +static int get_vring_base(struct vhost_dev *dev,
> +                          struct vhost_vring_state *ring,
> +                          bool skip_drain)
>  {
>      int ret;
> +    int request = VHOST_USER_GET_VRING_BASE;
> +    if (skip_drain) {
> +        request = VHOST_USER_GET_VRING_BASE_SKIP_DRAIN;
> +    }
> +
>      VhostUserMsg msg = {
> -        .hdr.request = VHOST_USER_GET_VRING_BASE,
> +        .hdr.request = request,
>          .hdr.flags = VHOST_USER_VERSION,
>          .payload.state = *ring,
>          .hdr.size = sizeof(msg.payload.state),
> @@ -1424,9 +1432,9 @@ static int vhost_user_get_vring_base(struct vhost_dev *dev,
>          return ret;
>      }
>  
> -    if (msg.hdr.request != VHOST_USER_GET_VRING_BASE) {
> +    if (msg.hdr.request != request) {
>          error_report("Received unexpected msg type. Expected %d received %d",
> -                     VHOST_USER_GET_VRING_BASE, msg.hdr.request);
> +                     request, msg.hdr.request);
>          return -EPROTO;
>      }
>  
> @@ -1440,6 +1448,25 @@ static int vhost_user_get_vring_base(struct vhost_dev *dev,
>      return 0;
>  }
>  
> +static int vhost_user_get_vring_base(struct vhost_dev *dev,
> +                                     struct vhost_vring_state *ring)
> +{
> +    return get_vring_base(dev, ring, false);
> +}
> +
> +static int vhost_user_get_vring_base_skip_drain(struct vhost_dev *dev,
> +                                     struct vhost_vring_state *ring)
> +{
> +    bool skip_drain_supported = vhost_user_has_protocol_feature(dev,
> +                               VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN);
> +
> +    if (!skip_drain_supported) {
> +        return 0;
> +    }
> +
> +    return get_vring_base(dev, ring, true);
> +}
> +
>  static int vhost_set_vring_file(struct vhost_dev *dev,
>                                  VhostUserRequest request,
>                                  struct vhost_vring_file *file)
> @@ -2551,6 +2578,12 @@ static int vhost_user_backend_init(struct vhost_dev *dev, void *opaque,
>                                 VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
>          }
>  
> +        if (!virtio_has_feature(protocol_features,
> +                                VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT)) {
> +            protocol_features &= ~(1ULL <<
> +                             VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN);
> +        }
> +
>          /* final set of protocol features */
>          u->protocol_features = protocol_features;
>          err = vhost_user_set_protocol_features(dev, u->protocol_features);
> @@ -3409,6 +3442,7 @@ const VhostOps user_ops = {
>          .vhost_set_vring_num = vhost_user_set_vring_num,
>          .vhost_set_vring_base = vhost_user_set_vring_base,
>          .vhost_get_vring_base = vhost_user_get_vring_base,
> +        .vhost_get_vring_base_skip_drain = vhost_user_get_vring_base_skip_drain,
>          .vhost_set_vring_kick = vhost_user_set_vring_kick,
>          .vhost_set_vring_call = vhost_user_set_vring_call,
>          .vhost_set_vring_err = vhost_user_set_vring_err,
> diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
> index 2a62550222..2c7f464d55 100644
> --- a/hw/virtio/vhost.c
> +++ b/hw/virtio/vhost.c
> @@ -1502,7 +1502,7 @@ static int do_vhost_virtqueue_stop(struct vhost_dev *dev,
>  {
>      if (skip_drain) {
>          assert(vhost_user_has_protocol_feature(dev,
> -               VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT));
> +               VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN));
>      }
>      int vhost_vq_index = dev->vhost_ops->vhost_get_vq_index(dev, idx);
>      struct vhost_vring_state state = {
> diff --git a/include/hw/virtio/vhost-backend.h b/include/hw/virtio/vhost-backend.h
> index d878d7b733..daa979a1aa 100644
> --- a/include/hw/virtio/vhost-backend.h
> +++ b/include/hw/virtio/vhost-backend.h
> @@ -190,6 +190,7 @@ typedef struct VhostOps {
>      vhost_set_vring_num_op vhost_set_vring_num;
>      vhost_set_vring_base_op vhost_set_vring_base;
>      vhost_get_vring_base_op vhost_get_vring_base;
> +    vhost_get_vring_base_op vhost_get_vring_base_skip_drain;
>      vhost_set_vring_kick_op vhost_set_vring_kick;
>      vhost_set_vring_call_op vhost_set_vring_call;
>      vhost_set_vring_err_op vhost_set_vring_err;
> diff --git a/include/hw/virtio/vhost-user.h b/include/hw/virtio/vhost-user.h
> index 06c360af18..47c13f8677 100644
> --- a/include/hw/virtio/vhost-user.h
> +++ b/include/hw/virtio/vhost-user.h
> @@ -36,6 +36,7 @@ enum VhostUserProtocolFeature {
>      VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT = 20,
>      VHOST_USER_PROTOCOL_F_GPA_ADDRESSES = 21,
>      VHOST_USER_PROTOCOL_F_SHMEM = 22,
> +    VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN = 23,
>      VHOST_USER_PROTOCOL_F_MAX
>  };
>  
> -- 
> 2.34.1

[PATCH v5 3/6] vhost-user: use skip_drain with GET_VRING_BASE_SKIP_DRAIN message

[Alexandr Moshkov]

Wire up the skip_drain parameter introduced in the previous commit to
the new GET_VRING_BASE_SKIP_DRAIN message. When skip_drain is set,
send GET_VRING_BASE_SKIP_DRAIN instead of GET_VRING_BASE, instructing
the back-end to suspend in-flight I/O immediately.

Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
---
 hw/virtio/vhost.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index 2c7f464d55..bd079f9080 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -1518,7 +1518,11 @@ static int do_vhost_virtqueue_stop(struct vhost_dev *dev,
     }
 
     if (!force) {
-        r = dev->vhost_ops->vhost_get_vring_base(dev, &state);
+        if (!skip_drain) {
+            r = dev->vhost_ops->vhost_get_vring_base(dev, &state);
+        } else {
+            r = dev->vhost_ops->vhost_get_vring_base_skip_drain(dev, &state);
+        }
         if (r < 0) {
             VHOST_OPS_DEBUG(r, "vhost VQ %u ring restore failed: %d", idx, r);
         }
-- 
2.34.1

[PATCH v5 4/6] vhost-user-blk: make inflight-migration prop mutable

[Alexandr Moshkov]

When migrating from a QEMU version that supports inflight-migration to
an older one that does not, there is no way to disable the feature at
runtime — the VM must be stopped and reconfigured. This is impractical
in production environments.

Make the inflight-migration property mutable after device realization
so it can be toggled via qom-set without restarting the VM.

Acked-by: Raphael Norwitz <rnorwitz@nvidia.com>
Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
---
 hw/block/vhost-user-blk.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
index e3b873af7c..650c004bde 100644
--- a/hw/block/vhost-user-blk.c
+++ b/hw/block/vhost-user-blk.c
@@ -619,6 +619,8 @@ static const VMStateDescription vmstate_vhost_user_blk = {
     }
 };
 
+static PropertyInfo vhost_user_blk_inflight_migration_prop;
+
 static const Property vhost_user_blk_properties[] = {
     DEFINE_PROP_CHR("chardev", VHostUserBlk, chardev),
     DEFINE_PROP_UINT16("num-queues", VHostUserBlk, num_queues,
@@ -632,8 +634,9 @@ static const Property vhost_user_blk_properties[] = {
                       VIRTIO_BLK_F_WRITE_ZEROES, true),
     DEFINE_PROP_BOOL("skip-get-vring-base-on-force-shutdown", VHostUserBlk,
                      skip_get_vring_base_on_force_shutdown, false),
-    DEFINE_PROP_BOOL("inflight-migration", VHostUserBlk,
-                     inflight_migration, false),
+    DEFINE_PROP("inflight-migration", VHostUserBlk, inflight_migration,
+                vhost_user_blk_inflight_migration_prop, bool,
+                .set_default = true, .defval.u = false),
 };
 
 static void vhost_user_blk_class_init(ObjectClass *klass, const void *data)
@@ -665,6 +668,9 @@ static const TypeInfo vhost_user_blk_info = {
 
 static void virtio_register_types(void)
 {
+    vhost_user_blk_inflight_migration_prop = qdev_prop_bool;
+    vhost_user_blk_inflight_migration_prop.realized_set_allowed = true;
+
     type_register_static(&vhost_user_blk_info);
 }
 
-- 
2.34.1

Re: [PATCH v5 4/6] vhost-user-blk: make inflight-migration prop mutable

[Michael S. Tsirkin]

On Tue, Jul 28, 2026 at 03:08:39PM +0500, Alexandr Moshkov wrote:
> When migrating from a QEMU version that supports inflight-migration to
> an older one that does not, there is no way to disable the feature at
> runtime — the VM must be stopped and reconfigured. This is impractical
> in production environments.
> 
> Make the inflight-migration property mutable after device realization
> so it can be toggled via qom-set without restarting the VM.
> 
> Acked-by: Raphael Norwitz <rnorwitz@nvidia.com>
> Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>



I think I am beginning to understand.


You are running qemu with inflight-migration on and want to migrate
to qemu without inflight-migration at all.


Since it is guest transparent you could retrofit it like this.

My question is why is it worth it, we do not normally support
migrating between qemu versions with different command lines.

> ---
>  hw/block/vhost-user-blk.c | 10 ++++++++--
>  1 file changed, 8 insertions(+), 2 deletions(-)
> 
> diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
> index e3b873af7c..650c004bde 100644
> --- a/hw/block/vhost-user-blk.c
> +++ b/hw/block/vhost-user-blk.c
> @@ -619,6 +619,8 @@ static const VMStateDescription vmstate_vhost_user_blk = {
>      }
>  };
>  
> +static PropertyInfo vhost_user_blk_inflight_migration_prop;
> +
>  static const Property vhost_user_blk_properties[] = {
>      DEFINE_PROP_CHR("chardev", VHostUserBlk, chardev),
>      DEFINE_PROP_UINT16("num-queues", VHostUserBlk, num_queues,
> @@ -632,8 +634,9 @@ static const Property vhost_user_blk_properties[] = {
>                        VIRTIO_BLK_F_WRITE_ZEROES, true),
>      DEFINE_PROP_BOOL("skip-get-vring-base-on-force-shutdown", VHostUserBlk,
>                       skip_get_vring_base_on_force_shutdown, false),
> -    DEFINE_PROP_BOOL("inflight-migration", VHostUserBlk,
> -                     inflight_migration, false),
> +    DEFINE_PROP("inflight-migration", VHostUserBlk, inflight_migration,
> +                vhost_user_blk_inflight_migration_prop, bool,
> +                .set_default = true, .defval.u = false),
>  };
>  
>  static void vhost_user_blk_class_init(ObjectClass *klass, const void *data)
> @@ -665,6 +668,9 @@ static const TypeInfo vhost_user_blk_info = {
>  
>  static void virtio_register_types(void)
>  {
> +    vhost_user_blk_inflight_migration_prop = qdev_prop_bool;
> +    vhost_user_blk_inflight_migration_prop.realized_set_allowed = true;
> +
>      type_register_static(&vhost_user_blk_info);
>  }

So then,  for example, let us say I paused the VM, then set the flag,
now inflight is on but GET_BASE did not drain it?

> -- 
> 2.34.1

Re: [PATCH v5 4/6] vhost-user-blk: make inflight-migration prop mutable

[Michael S. Tsirkin]

On Wed, Jul 29, 2026 at 03:09:57PM +0500, Alexandr Moshkov wrote:
> 
> On 7/29/26 14:32, Michael S. Tsirkin wrote:
> 
>     On Tue, Jul 28, 2026 at 03:08:39PM +0500, Alexandr Moshkov wrote:
> 
>         When migrating from a QEMU version that supports inflight-migration to
>         an older one that does not, there is no way to disable the feature at
>         runtime — the VM must be stopped and reconfigured. This is impractical
>         in production environments.
> 
>         Make the inflight-migration property mutable after device realization
>         so it can be toggled via qom-set without restarting the VM.
> 
>         Acked-by: Raphael Norwitz <rnorwitz@nvidia.com>
>         Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
> 
> 
> 
>     I think I am beginning to understand.
> 
> 
>     You are running qemu with inflight-migration on and want to migrate
>     to qemu without inflight-migration at all.
> 
> 
>     Since it is guest transparent you could retrofit it like this.
> 
>     My question is why is it worth it, we do not normally support
>     migrating between qemu versions with different command lines.
> 
> I think you right. Maybe I've been focusing too much on the ability to migrate 
> between versions of qemu with and without inflight migration.
> 
> This series allows to turn off and on the inflight-migration feature at runtime
> without recreating the VM (it was the only way to turn off the feature, since
> the protocol feature could no longer be turned off, after initialization with
> the backend). This is more important feature, and it leads to the fact that
> this feature allows to migrate between different versions of qemu (with and
> without inflight-migration support).
> 
>     
> 
>         ---
>          hw/block/vhost-user-blk.c | 10 ++++++++--
>          1 file changed, 8 insertions(+), 2 deletions(-)
> 
>         diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
>         index e3b873af7c..650c004bde 100644
>         --- a/hw/block/vhost-user-blk.c
>         +++ b/hw/block/vhost-user-blk.c
>         @@ -619,6 +619,8 @@ static const VMStateDescription vmstate_vhost_user_blk = {
>              }
>          };
> 
>         +static PropertyInfo vhost_user_blk_inflight_migration_prop;
>         +
>          static const Property vhost_user_blk_properties[] = {
>              DEFINE_PROP_CHR("chardev", VHostUserBlk, chardev),
>              DEFINE_PROP_UINT16("num-queues", VHostUserBlk, num_queues,
>         @@ -632,8 +634,9 @@ static const Property vhost_user_blk_properties[] = {
>                                VIRTIO_BLK_F_WRITE_ZEROES, true),
>              DEFINE_PROP_BOOL("skip-get-vring-base-on-force-shutdown", VHostUserBlk,
>                               skip_get_vring_base_on_force_shutdown, false),
>         -    DEFINE_PROP_BOOL("inflight-migration", VHostUserBlk,
>         -                     inflight_migration, false),
>         +    DEFINE_PROP("inflight-migration", VHostUserBlk, inflight_migration,
>         +                vhost_user_blk_inflight_migration_prop, bool,
>         +                .set_default = true, .defval.u = false),
>          };
> 
>          static void vhost_user_blk_class_init(ObjectClass *klass, const void *data)
>         @@ -665,6 +668,9 @@ static const TypeInfo vhost_user_blk_info = {
> 
>          static void virtio_register_types(void)
>          {
>         +    vhost_user_blk_inflight_migration_prop = qdev_prop_bool;
>         +    vhost_user_blk_inflight_migration_prop.realized_set_allowed = true;
>         +
>              type_register_static(&vhost_user_blk_info);
>          }
> 
>     So then,  for example, let us say I paused the VM, then set the flag,
>     now inflight is on but GET_BASE did not drain it?
> 
> If I understood the question correctly, this is valid behavior. Before
> migration QEMU check protocol features to understand does the backend support
> inflight migration. If it does, after that QEMU migrate inflight buffer to
> other VM. If it's not, return error before migration started.  


I apologise i reverted the logic in the question.


I start vm and inflight is on.
I stop vm get base does not drain.
i turn inflight off.
now it looks like it will happily migrate?

>         --
>         2.34.1
> 

Re: [PATCH v5 4/6] vhost-user-blk: make inflight-migration prop mutable

[Alexandr Moshkov]

On 7/29/26 15:14, Michael S. Tsirkin wrote:
> On Wed, Jul 29, 2026 at 03:09:57PM +0500, Alexandr Moshkov wrote:
>> On 7/29/26 14:32, Michael S. Tsirkin wrote:
>>
>>      On Tue, Jul 28, 2026 at 03:08:39PM +0500, Alexandr Moshkov wrote:
>>
>>          When migrating from a QEMU version that supports inflight-migration to
>>          an older one that does not, there is no way to disable the feature at
>>          runtime — the VM must be stopped and reconfigured. This is impractical
>>          in production environments.
>>
>>          Make the inflight-migration property mutable after device realization
>>          so it can be toggled via qom-set without restarting the VM.
>>
>>          Acked-by: Raphael Norwitz <rnorwitz@nvidia.com>
>>          Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
>>
>>
>>
>>      I think I am beginning to understand.
>>
>>
>>      You are running qemu with inflight-migration on and want to migrate
>>      to qemu without inflight-migration at all.
>>
>>
>>      Since it is guest transparent you could retrofit it like this.
>>
>>      My question is why is it worth it, we do not normally support
>>      migrating between qemu versions with different command lines.
>>
>> I think you right. Maybe I've been focusing too much on the ability to migrate
>> between versions of qemu with and without inflight migration.
>>
>> This series allows to turn off and on the inflight-migration feature at runtime
>> without recreating the VM (it was the only way to turn off the feature, since
>> the protocol feature could no longer be turned off, after initialization with
>> the backend). This is more important feature, and it leads to the fact that
>> this feature allows to migrate between different versions of qemu (with and
>> without inflight-migration support).
>>
>>      
>>
>>          ---
>>           hw/block/vhost-user-blk.c | 10 ++++++++--
>>           1 file changed, 8 insertions(+), 2 deletions(-)
>>
>>          diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
>>          index e3b873af7c..650c004bde 100644
>>          --- a/hw/block/vhost-user-blk.c
>>          +++ b/hw/block/vhost-user-blk.c
>>          @@ -619,6 +619,8 @@ static const VMStateDescription vmstate_vhost_user_blk = {
>>               }
>>           };
>>
>>          +static PropertyInfo vhost_user_blk_inflight_migration_prop;
>>          +
>>           static const Property vhost_user_blk_properties[] = {
>>               DEFINE_PROP_CHR("chardev", VHostUserBlk, chardev),
>>               DEFINE_PROP_UINT16("num-queues", VHostUserBlk, num_queues,
>>          @@ -632,8 +634,9 @@ static const Property vhost_user_blk_properties[] = {
>>                                 VIRTIO_BLK_F_WRITE_ZEROES, true),
>>               DEFINE_PROP_BOOL("skip-get-vring-base-on-force-shutdown", VHostUserBlk,
>>                                skip_get_vring_base_on_force_shutdown, false),
>>          -    DEFINE_PROP_BOOL("inflight-migration", VHostUserBlk,
>>          -                     inflight_migration, false),
>>          +    DEFINE_PROP("inflight-migration", VHostUserBlk, inflight_migration,
>>          +                vhost_user_blk_inflight_migration_prop, bool,
>>          +                .set_default = true, .defval.u = false),
>>           };
>>
>>           static void vhost_user_blk_class_init(ObjectClass *klass, const void *data)
>>          @@ -665,6 +668,9 @@ static const TypeInfo vhost_user_blk_info = {
>>
>>           static void virtio_register_types(void)
>>           {
>>          +    vhost_user_blk_inflight_migration_prop = qdev_prop_bool;
>>          +    vhost_user_blk_inflight_migration_prop.realized_set_allowed = true;
>>          +
>>               type_register_static(&vhost_user_blk_info);
>>           }
>>
>>      So then,  for example, let us say I paused the VM, then set the flag,
>>      now inflight is on but GET_BASE did not drain it?
>>
>> If I understood the question correctly, this is valid behavior. Before
>> migration QEMU check protocol features to understand does the backend support
>> inflight migration. If it does, after that QEMU migrate inflight buffer to
>> other VM. If it's not, return error before migration started.
>
> I apologise i reverted the logic in the question.
>
>
> I start vm and inflight is on.
> I stop vm get base does not drain.

In case of VM stop, backend wait to drain all requests. Ability to 
perform drain or not available only during migration by skip_drain 
variable in vhost_user_blk_stop().
skip_drain true only if inflight_migration is on and runstate is 
FINISH_MIGRATION.

> i turn inflight off.
> now it looks like it will happily migrate?

So in this case, when vm migrated with inflight off, it will leads to 
using GET_VRING_BASE message, that wait all requests to be drained.

>
>>          --
>>          2.34.1
>>

Re: [PATCH v5 4/6] vhost-user-blk: make inflight-migration prop mutable

[Michael S. Tsirkin]

On Wed, Jul 29, 2026 at 03:35:31PM +0500, Alexandr Moshkov wrote:
> 
> On 7/29/26 15:14, Michael S. Tsirkin wrote:
> > On Wed, Jul 29, 2026 at 03:09:57PM +0500, Alexandr Moshkov wrote:
> > > On 7/29/26 14:32, Michael S. Tsirkin wrote:
> > > 
> > >      On Tue, Jul 28, 2026 at 03:08:39PM +0500, Alexandr Moshkov wrote:
> > > 
> > >          When migrating from a QEMU version that supports inflight-migration to
> > >          an older one that does not, there is no way to disable the feature at
> > >          runtime — the VM must be stopped and reconfigured. This is impractical
> > >          in production environments.
> > > 
> > >          Make the inflight-migration property mutable after device realization
> > >          so it can be toggled via qom-set without restarting the VM.
> > > 
> > >          Acked-by: Raphael Norwitz <rnorwitz@nvidia.com>
> > >          Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
> > > 
> > > 
> > > 
> > >      I think I am beginning to understand.
> > > 
> > > 
> > >      You are running qemu with inflight-migration on and want to migrate
> > >      to qemu without inflight-migration at all.
> > > 
> > > 
> > >      Since it is guest transparent you could retrofit it like this.
> > > 
> > >      My question is why is it worth it, we do not normally support
> > >      migrating between qemu versions with different command lines.
> > > 
> > > I think you right. Maybe I've been focusing too much on the ability to migrate
> > > between versions of qemu with and without inflight migration.
> > > 
> > > This series allows to turn off and on the inflight-migration feature at runtime
> > > without recreating the VM (it was the only way to turn off the feature, since
> > > the protocol feature could no longer be turned off, after initialization with
> > > the backend). This is more important feature, and it leads to the fact that
> > > this feature allows to migrate between different versions of qemu (with and
> > > without inflight-migration support).
> > > 
> > > 
> > >          ---
> > >           hw/block/vhost-user-blk.c | 10 ++++++++--
> > >           1 file changed, 8 insertions(+), 2 deletions(-)
> > > 
> > >          diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
> > >          index e3b873af7c..650c004bde 100644
> > >          --- a/hw/block/vhost-user-blk.c
> > >          +++ b/hw/block/vhost-user-blk.c
> > >          @@ -619,6 +619,8 @@ static const VMStateDescription vmstate_vhost_user_blk = {
> > >               }
> > >           };
> > > 
> > >          +static PropertyInfo vhost_user_blk_inflight_migration_prop;
> > >          +
> > >           static const Property vhost_user_blk_properties[] = {
> > >               DEFINE_PROP_CHR("chardev", VHostUserBlk, chardev),
> > >               DEFINE_PROP_UINT16("num-queues", VHostUserBlk, num_queues,
> > >          @@ -632,8 +634,9 @@ static const Property vhost_user_blk_properties[] = {
> > >                                 VIRTIO_BLK_F_WRITE_ZEROES, true),
> > >               DEFINE_PROP_BOOL("skip-get-vring-base-on-force-shutdown", VHostUserBlk,
> > >                                skip_get_vring_base_on_force_shutdown, false),
> > >          -    DEFINE_PROP_BOOL("inflight-migration", VHostUserBlk,
> > >          -                     inflight_migration, false),
> > >          +    DEFINE_PROP("inflight-migration", VHostUserBlk, inflight_migration,
> > >          +                vhost_user_blk_inflight_migration_prop, bool,
> > >          +                .set_default = true, .defval.u = false),
> > >           };
> > > 
> > >           static void vhost_user_blk_class_init(ObjectClass *klass, const void *data)
> > >          @@ -665,6 +668,9 @@ static const TypeInfo vhost_user_blk_info = {
> > > 
> > >           static void virtio_register_types(void)
> > >           {
> > >          +    vhost_user_blk_inflight_migration_prop = qdev_prop_bool;
> > >          +    vhost_user_blk_inflight_migration_prop.realized_set_allowed = true;
> > >          +
> > >               type_register_static(&vhost_user_blk_info);
> > >           }
> > > 
> > >      So then,  for example, let us say I paused the VM, then set the flag,
> > >      now inflight is on but GET_BASE did not drain it?
> > > 
> > > If I understood the question correctly, this is valid behavior. Before
> > > migration QEMU check protocol features to understand does the backend support
> > > inflight migration. If it does, after that QEMU migrate inflight buffer to
> > > other VM. If it's not, return error before migration started.
> > 
> > I apologise i reverted the logic in the question.
> > 
> > 
> > I start vm and inflight is on.
> > I stop vm get base does not drain.
> 
> In case of VM stop, backend wait to drain all requests. Ability to perform
> drain or not available only during migration by skip_drain variable in
> vhost_user_blk_stop().

What if it was migration but it failed?

> skip_drain true only if inflight_migration is on and runstate is
> FINISH_MIGRATION.

This btw I don't much like, a stopped VM would preferably
behave the same whatever the reason to stop.

> 
> > i turn inflight off.
> > now it looks like it will happily migrate?
> 
> So in this case, when vm migrated with inflight off, it will leads to using
> GET_VRING_BASE message, that wait all requests to be drained.
> 
> > 
> > >          --
> > >          2.34.1
> > > 

Re: [PATCH v5 4/6] vhost-user-blk: make inflight-migration prop mutable

[Michael S. Tsirkin]

On Wed, Jul 29, 2026 at 03:56:54PM +0500, Alexandr Moshkov wrote:
> 
> On 7/29/26 15:48, Michael S. Tsirkin wrote:
> 
>     On Wed, Jul 29, 2026 at 03:35:31PM +0500, Alexandr Moshkov wrote:
> 
>         On 7/29/26 15:14, Michael S. Tsirkin wrote:
> 
>             On Wed, Jul 29, 2026 at 03:09:57PM +0500, Alexandr Moshkov wrote:
> 
>                 On 7/29/26 14:32, Michael S. Tsirkin wrote:
> 
>                      On Tue, Jul 28, 2026 at 03:08:39PM +0500, Alexandr Moshkov wrote:
> 
>                          When migrating from a QEMU version that supports inflight-migration to
>                          an older one that does not, there is no way to disable the feature at
>                          runtime — the VM must be stopped and reconfigured. This is impractical
>                          in production environments.
> 
>                          Make the inflight-migration property mutable after device realization
>                          so it can be toggled via qom-set without restarting the VM.
> 
>                          Acked-by: Raphael Norwitz <rnorwitz@nvidia.com>
>                          Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
> 
> 
> 
>                      I think I am beginning to understand.
> 
> 
>                      You are running qemu with inflight-migration on and want to migrate
>                      to qemu without inflight-migration at all.
> 
> 
>                      Since it is guest transparent you could retrofit it like this.
> 
>                      My question is why is it worth it, we do not normally support
>                      migrating between qemu versions with different command lines.
> 
>                 I think you right. Maybe I've been focusing too much on the ability to migrate
>                 between versions of qemu with and without inflight migration.
> 
>                 This series allows to turn off and on the inflight-migration feature at runtime
>                 without recreating the VM

restarting you mean.

> (it was the only way to turn off the feature, since
>                 the protocol feature could no longer be turned off, after initialization with
>                 the backend). This is more important feature, and it leads to the fact that
>                 this feature allows to migrate between different versions of qemu (with and
>                 without inflight-migration support).

so again it's not the 1st feature we have like this. we tie migration
to the machine type and same set of command line flags
specifically to keep things manageable.

really cross version migration is a pain as it is.

> 
>                          ---
>                           hw/block/vhost-user-blk.c | 10 ++++++++--
>                           1 file changed, 8 insertions(+), 2 deletions(-)
> 
>                          diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
>                          index e3b873af7c..650c004bde 100644
>                          --- a/hw/block/vhost-user-blk.c
>                          +++ b/hw/block/vhost-user-blk.c
>                          @@ -619,6 +619,8 @@ static const VMStateDescription vmstate_vhost_user_blk = {
>                               }
>                           };
> 
>                          +static PropertyInfo vhost_user_blk_inflight_migration_prop;
>                          +
>                           static const Property vhost_user_blk_properties[] = {
>                               DEFINE_PROP_CHR("chardev", VHostUserBlk, chardev),
>                               DEFINE_PROP_UINT16("num-queues", VHostUserBlk, num_queues,
>                          @@ -632,8 +634,9 @@ static const Property vhost_user_blk_properties[] = {
>                                                 VIRTIO_BLK_F_WRITE_ZEROES, true),
>                               DEFINE_PROP_BOOL("skip-get-vring-base-on-force-shutdown", VHostUserBlk,
>                                                skip_get_vring_base_on_force_shutdown, false),
>                          -    DEFINE_PROP_BOOL("inflight-migration", VHostUserBlk,
>                          -                     inflight_migration, false),
>                          +    DEFINE_PROP("inflight-migration", VHostUserBlk, inflight_migration,
>                          +                vhost_user_blk_inflight_migration_prop, bool,
>                          +                .set_default = true, .defval.u = false),
>                           };
> 
>                           static void vhost_user_blk_class_init(ObjectClass *klass, const void *data)
>                          @@ -665,6 +668,9 @@ static const TypeInfo vhost_user_blk_info = {
> 
>                           static void virtio_register_types(void)
>                           {
>                          +    vhost_user_blk_inflight_migration_prop = qdev_prop_bool;
>                          +    vhost_user_blk_inflight_migration_prop.realized_set_allowed = true;
>                          +
>                               type_register_static(&vhost_user_blk_info);
>                           }
> 
>                      So then,  for example, let us say I paused the VM, then set the flag,
>                      now inflight is on but GET_BASE did not drain it?
> 
>                 If I understood the question correctly, this is valid behavior. Before
>                 migration QEMU check protocol features to understand does the backend support
>                 inflight migration. If it does, after that QEMU migrate inflight buffer to
>                 other VM. If it's not, return error before migration started.
> 
>             I apologise i reverted the logic in the question.
> 
> 
>             I start vm and inflight is on.
>             I stop vm get base does not drain.
> 
>         In case of VM stop, backend wait to drain all requests. Ability to perform
>         drain or not available only during migration by skip_drain variable in
>         vhost_user_blk_stop().
> 
>     What if it was migration but it failed?
> 
> If migration failed, source vm just keep using inflight region, backend
> continue to execute inflight requests that was tried to migrate.


Sorry, to be more clear. I mean migration succeeded but destination failed to
start, so now vm is stopped and we are now trying migrating to a different destination.

IIUC currently on VM stop we simply send GET_BASE and this stops
backend, and depending on features things remain in the inflight buffer.
Correct me if  I am wrong.

I think that we do not want a slow flush on vm stop if we
can avoid it, and we also do not want backend to keep changing
guest memory when VM is stopped. No?




> 
> 
>         skip_drain true only if inflight_migration is on and runstate is
>         FINISH_MIGRATION.

FINISH_MIGRATE? actually i do not see where it affects it.

> 
>     This btw I don't much like, a stopped VM would preferably
>     behave the same whatever the reason to stop.
> 
> Well, I don't see any other way to implement this.


I am not 100% sure it's implementable as described, but I guess
you could block changing the property if VM and thus the backend
is not connected and running.


Which is even more complexity but at least it is consistent.






> I think backend must perform
> drain in all cases except live migration with all the necessary checks 
> (protocol features, special message GET_VRING_BASE_SKIP_DRAIN).


This is not what is going on now, right? Whether it drains depends
on protocol features not on VM state and I think we should
keep it like this.



> 
> 
>             i turn inflight off.
>             now it looks like it will happily migrate?
> 
>         So in this case, when vm migrated with inflight off, it will leads to using
>         GET_VRING_BASE message, that wait all requests to be drained.
> 
> 
>                          --
>                          2.34.1
> 
>


i mean it migrated with on. vm was stopped then it flipped to off.






-- 
MST 

Re: [PATCH v5 4/6] vhost-user-blk: make inflight-migration prop mutable

[Alexandr Moshkov]

On 7/29/26 17:13, Michael S. Tsirkin wrote:
> On Wed, Jul 29, 2026 at 03:56:54PM +0500, Alexandr Moshkov wrote:
>> On 7/29/26 15:48, Michael S. Tsirkin wrote:
>>
>>      On Wed, Jul 29, 2026 at 03:35:31PM +0500, Alexandr Moshkov wrote:
>>
>>          On 7/29/26 15:14, Michael S. Tsirkin wrote:
>>
>>              On Wed, Jul 29, 2026 at 03:09:57PM +0500, Alexandr Moshkov wrote:
>>
>>                  On 7/29/26 14:32, Michael S. Tsirkin wrote:
>>
>>                       On Tue, Jul 28, 2026 at 03:08:39PM +0500, Alexandr Moshkov wrote:
>>
>>                           When migrating from a QEMU version that supports inflight-migration to
>>                           an older one that does not, there is no way to disable the feature at
>>                           runtime — the VM must be stopped and reconfigured. This is impractical
>>                           in production environments.
>>
>>                           Make the inflight-migration property mutable after device realization
>>                           so it can be toggled via qom-set without restarting the VM.
>>
>>                           Acked-by: Raphael Norwitz <rnorwitz@nvidia.com>
>>                           Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
>>
>>
>>
>>                       I think I am beginning to understand.
>>
>>
>>                       You are running qemu with inflight-migration on and want to migrate
>>                       to qemu without inflight-migration at all.
>>
>>
>>                       Since it is guest transparent you could retrofit it like this.
>>
>>                       My question is why is it worth it, we do not normally support
>>                       migrating between qemu versions with different command lines.
>>
>>                  I think you right. Maybe I've been focusing too much on the ability to migrate
>>                  between versions of qemu with and without inflight migration.
>>
>>                  This series allows to turn off and on the inflight-migration feature at runtime
>>                  without recreating the VM
> restarting you mean.
Yes, sorry.
>
>> (it was the only way to turn off the feature, since
>>                  the protocol feature could no longer be turned off, after initialization with
>>                  the backend). This is more important feature, and it leads to the fact that
>>                  this feature allows to migrate between different versions of qemu (with and
>>                  without inflight-migration support).
> so again it's not the 1st feature we have like this. we tie migration
> to the machine type and same set of command line flags
> specifically to keep things manageable.
>
> really cross version migration is a pain as it is.

I understand that, thanks! Maybe we should focus on the fact that this 
series allows to turn inflight_migration on and off (without cross 
version migration) ?

Firstly, I wanted to implement this through migration capabilities, but 
in one of the threads,
it was agreed that a separate capability only for vhost-user-blk is not 
a good idea.
Therefore, I implemented it through a parameter that can be changed 
using qom-set.

>
>>                           ---
>>                            hw/block/vhost-user-blk.c | 10 ++++++++--
>>                            1 file changed, 8 insertions(+), 2 deletions(-)
>>
>>                           diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
>>                           index e3b873af7c..650c004bde 100644
>>                           --- a/hw/block/vhost-user-blk.c
>>                           +++ b/hw/block/vhost-user-blk.c
>>                           @@ -619,6 +619,8 @@ static const VMStateDescription vmstate_vhost_user_blk = {
>>                                }
>>                            };
>>
>>                           +static PropertyInfo vhost_user_blk_inflight_migration_prop;
>>                           +
>>                            static const Property vhost_user_blk_properties[] = {
>>                                DEFINE_PROP_CHR("chardev", VHostUserBlk, chardev),
>>                                DEFINE_PROP_UINT16("num-queues", VHostUserBlk, num_queues,
>>                           @@ -632,8 +634,9 @@ static const Property vhost_user_blk_properties[] = {
>>                                                  VIRTIO_BLK_F_WRITE_ZEROES, true),
>>                                DEFINE_PROP_BOOL("skip-get-vring-base-on-force-shutdown", VHostUserBlk,
>>                                                 skip_get_vring_base_on_force_shutdown, false),
>>                           -    DEFINE_PROP_BOOL("inflight-migration", VHostUserBlk,
>>                           -                     inflight_migration, false),
>>                           +    DEFINE_PROP("inflight-migration", VHostUserBlk, inflight_migration,
>>                           +                vhost_user_blk_inflight_migration_prop, bool,
>>                           +                .set_default = true, .defval.u = false),
>>                            };
>>
>>                            static void vhost_user_blk_class_init(ObjectClass *klass, const void *data)
>>                           @@ -665,6 +668,9 @@ static const TypeInfo vhost_user_blk_info = {
>>
>>                            static void virtio_register_types(void)
>>                            {
>>                           +    vhost_user_blk_inflight_migration_prop = qdev_prop_bool;
>>                           +    vhost_user_blk_inflight_migration_prop.realized_set_allowed = true;
>>                           +
>>                                type_register_static(&vhost_user_blk_info);
>>                            }
>>
>>                       So then,  for example, let us say I paused the VM, then set the flag,
>>                       now inflight is on but GET_BASE did not drain it?
>>
>>                  If I understood the question correctly, this is valid behavior. Before
>>                  migration QEMU check protocol features to understand does the backend support
>>                  inflight migration. If it does, after that QEMU migrate inflight buffer to
>>                  other VM. If it's not, return error before migration started.
>>
>>              I apologise i reverted the logic in the question.
>>
>>
>>              I start vm and inflight is on.
>>              I stop vm get base does not drain.
>>
>>          In case of VM stop, backend wait to drain all requests. Ability to perform
>>          drain or not available only during migration by skip_drain variable in
>>          vhost_user_blk_stop().
>>
>>      What if it was migration but it failed?
>>
>> If migration failed, source vm just keep using inflight region, backend
>> continue to execute inflight requests that was tried to migrate.
>
> Sorry, to be more clear. I mean migration succeeded but destination failed to
> start, so now vm is stopped and we are now trying migrating to a different destination.

Is this a real case? I was thinking in case of any destination errors 
(migration process or starting), source vm has to work...

I guess in that case, inflight requests can only be executed on 
different destination or in current destination if vm be able to start.

> IIUC currently on VM stop we simply send GET_BASE and this stops
> backend, and depending on features things remain in the inflight buffer.
> Correct me if  I am wrong.
Yes backend has to disconnect from QEMU.. So it not be changing guest 
memory.
>
> I think that we do not want a slow flush on vm stop if we
> can avoid it, and we also do not want backend to keep changing
> guest memory when VM is stopped. No?
Yes, it is
>>
>>          skip_drain true only if inflight_migration is on and runstate is
>>          FINISH_MIGRATION.
> FINISH_MIGRATE? actually i do not see where it affects it.

Sorry, I didn't understand) Who affects who?

>
>>      This btw I don't much like, a stopped VM would preferably
>>      behave the same whatever the reason to stop.
>>
>> Well, I don't see any other way to implement this.
>
> I am not 100% sure it's implementable as described, but I guess
> you could block changing the property if VM and thus the backend
> is not connected and running.
>
>
> Which is even more complexity but at least it is consistent.

Oh, I understand what you thinking about.

I agreed, but why implementation (in this series) is not consistent? I 
was thinking that current (in current qemu version) impl is not, because 
GET_VRING_BASE has two behaviors, depending on the presence of the 
protocol feature (drain or not drain requests).

Now, I think it would be more clear: one message for drain 
GET_VRING_BASE (as it was before), one for not drain 
GET_VRING_BASE_SKIP_DRAIN.

>
>
>
>
>
>
>> I think backend must perform
>> drain in all cases except live migration with all the necessary checks
>> (protocol features, special message GET_VRING_BASE_SKIP_DRAIN).
>
> This is not what is going on now, right? Whether it drains depends
> on protocol features not on VM state and I think we should
> keep it like this.

Well yes, now in current qemu version drain or not drain depends on 
protocol feature only, that try to be negotiated only if 
inflight_migration is set on.

In patch: in case live migration (if protocol feature and  device param) 
sent message GET_VRING_BASE_SKIP_DRAIN to skip drain, in other cases 
sent other message - GET_VRING_BASE with draining all requests.
So drain or not drain depends on message that QEMU send to backend.

>
>
>>
>>              i turn inflight off.
>>              now it looks like it will happily migrate?
>>
>>          So in this case, when vm migrated with inflight off, it will leads to using
>>          GET_VRING_BASE message, that wait all requests to be drained.
>>
>>
>>                           --
>>                           2.34.1
>>
>>
>
> i mean it migrated with on. vm was stopped then it flipped to off.
After stop vm will send GET_VRING_BASE to backend, that lead to drain 
all requests. Non of the inflight requests will be missed here, If 
that's what you mean.
>
>
>
>
>

Re: [PATCH v5 4/6] vhost-user-blk: make inflight-migration prop mutable

[Michael S. Tsirkin]

On Wed, Jul 29, 2026 at 07:07:11PM +0500, Alexandr Moshkov wrote:
> 
> On 7/29/26 17:13, Michael S. Tsirkin wrote:
> > On Wed, Jul 29, 2026 at 03:56:54PM +0500, Alexandr Moshkov wrote:
> > > On 7/29/26 15:48, Michael S. Tsirkin wrote:
> > > 
> > >      On Wed, Jul 29, 2026 at 03:35:31PM +0500, Alexandr Moshkov wrote:
> > > 
> > >          On 7/29/26 15:14, Michael S. Tsirkin wrote:
> > > 
> > >              On Wed, Jul 29, 2026 at 03:09:57PM +0500, Alexandr Moshkov wrote:
> > > 
> > >                  On 7/29/26 14:32, Michael S. Tsirkin wrote:
> > > 
> > >                       On Tue, Jul 28, 2026 at 03:08:39PM +0500, Alexandr Moshkov wrote:
> > > 
> > >                           When migrating from a QEMU version that supports inflight-migration to
> > >                           an older one that does not, there is no way to disable the feature at
> > >                           runtime — the VM must be stopped and reconfigured. This is impractical
> > >                           in production environments.
> > > 
> > >                           Make the inflight-migration property mutable after device realization
> > >                           so it can be toggled via qom-set without restarting the VM.
> > > 
> > >                           Acked-by: Raphael Norwitz <rnorwitz@nvidia.com>
> > >                           Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
> > > 
> > > 
> > > 
> > >                       I think I am beginning to understand.
> > > 
> > > 
> > >                       You are running qemu with inflight-migration on and want to migrate
> > >                       to qemu without inflight-migration at all.
> > > 
> > > 
> > >                       Since it is guest transparent you could retrofit it like this.
> > > 
> > >                       My question is why is it worth it, we do not normally support
> > >                       migrating between qemu versions with different command lines.
> > > 
> > >                  I think you right. Maybe I've been focusing too much on the ability to migrate
> > >                  between versions of qemu with and without inflight migration.
> > > 
> > >                  This series allows to turn off and on the inflight-migration feature at runtime
> > >                  without recreating the VM
> > restarting you mean.
> Yes, sorry.
> > 
> > > (it was the only way to turn off the feature, since
> > >                  the protocol feature could no longer be turned off, after initialization with
> > >                  the backend). This is more important feature, and it leads to the fact that
> > >                  this feature allows to migrate between different versions of qemu (with and
> > >                  without inflight-migration support).
> > so again it's not the 1st feature we have like this. we tie migration
> > to the machine type and same set of command line flags
> > specifically to keep things manageable.
> > 
> > really cross version migration is a pain as it is.
> 
> I understand that, thanks! Maybe we should focus on the fact that this
> series allows to turn inflight_migration on and off (without cross version
> migration) ?
> 
> Firstly, I wanted to implement this through migration capabilities, but in
> one of the threads,
> it was agreed that a separate capability only for vhost-user-blk is not a
> good idea.

I mean it's a generic thing.

> Therefore, I implemented it through a parameter that can be changed using
> qom-set.
> 
> > 
> > >                           ---
> > >                            hw/block/vhost-user-blk.c | 10 ++++++++--
> > >                            1 file changed, 8 insertions(+), 2 deletions(-)
> > > 
> > >                           diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
> > >                           index e3b873af7c..650c004bde 100644
> > >                           --- a/hw/block/vhost-user-blk.c
> > >                           +++ b/hw/block/vhost-user-blk.c
> > >                           @@ -619,6 +619,8 @@ static const VMStateDescription vmstate_vhost_user_blk = {
> > >                                }
> > >                            };
> > > 
> > >                           +static PropertyInfo vhost_user_blk_inflight_migration_prop;
> > >                           +
> > >                            static const Property vhost_user_blk_properties[] = {
> > >                                DEFINE_PROP_CHR("chardev", VHostUserBlk, chardev),
> > >                                DEFINE_PROP_UINT16("num-queues", VHostUserBlk, num_queues,
> > >                           @@ -632,8 +634,9 @@ static const Property vhost_user_blk_properties[] = {
> > >                                                  VIRTIO_BLK_F_WRITE_ZEROES, true),
> > >                                DEFINE_PROP_BOOL("skip-get-vring-base-on-force-shutdown", VHostUserBlk,
> > >                                                 skip_get_vring_base_on_force_shutdown, false),
> > >                           -    DEFINE_PROP_BOOL("inflight-migration", VHostUserBlk,
> > >                           -                     inflight_migration, false),
> > >                           +    DEFINE_PROP("inflight-migration", VHostUserBlk, inflight_migration,
> > >                           +                vhost_user_blk_inflight_migration_prop, bool,
> > >                           +                .set_default = true, .defval.u = false),
> > >                            };
> > > 
> > >                            static void vhost_user_blk_class_init(ObjectClass *klass, const void *data)
> > >                           @@ -665,6 +668,9 @@ static const TypeInfo vhost_user_blk_info = {
> > > 
> > >                            static void virtio_register_types(void)
> > >                            {
> > >                           +    vhost_user_blk_inflight_migration_prop = qdev_prop_bool;
> > >                           +    vhost_user_blk_inflight_migration_prop.realized_set_allowed = true;
> > >                           +
> > >                                type_register_static(&vhost_user_blk_info);
> > >                            }
> > > 
> > >                       So then,  for example, let us say I paused the VM, then set the flag,
> > >                       now inflight is on but GET_BASE did not drain it?
> > > 
> > >                  If I understood the question correctly, this is valid behavior. Before
> > >                  migration QEMU check protocol features to understand does the backend support
> > >                  inflight migration. If it does, after that QEMU migrate inflight buffer to
> > >                  other VM. If it's not, return error before migration started.
> > > 
> > >              I apologise i reverted the logic in the question.
> > > 
> > > 
> > >              I start vm and inflight is on.
> > >              I stop vm get base does not drain.
> > > 
> > >          In case of VM stop, backend wait to drain all requests. Ability to perform
> > >          drain or not available only during migration by skip_drain variable in
> > >          vhost_user_blk_stop().
> > > 
> > >      What if it was migration but it failed?
> > > 
> > > If migration failed, source vm just keep using inflight region, backend
> > > continue to execute inflight requests that was tried to migrate.
> > 
> > Sorry, to be more clear. I mean migration succeeded but destination failed to
> > start, so now vm is stopped and we are now trying migrating to a different destination.
> 
> Is this a real case? I was thinking in case of any destination errors
> (migration process or starting), source vm has to work...

It does not start automatically in all cases, no.

> I guess in that case, inflight requests can only be executed on different
> destination or in current destination if vm be able to start.
> 
> > IIUC currently on VM stop we simply send GET_BASE and this stops
> > backend, and depending on features things remain in the inflight buffer.
> > Correct me if  I am wrong.
> Yes backend has to disconnect from QEMU.. So it not be changing guest
> memory.
> > 
> > I think that we do not want a slow flush on vm stop if we
> > can avoid it, and we also do not want backend to keep changing
> > guest memory when VM is stopped. No?
> Yes, it is
> > > 
> > >          skip_drain true only if inflight_migration is on and runstate is
> > >          FINISH_MIGRATION.
> > FINISH_MIGRATE? actually i do not see where it affects it.
> 
> Sorry, I didn't understand) Who affects who?


> > 
> > >      This btw I don't much like, a stopped VM would preferably
> > >      behave the same whatever the reason to stop.
> > > 
> > > Well, I don't see any other way to implement this.
> > 
> > I am not 100% sure it's implementable as described, but I guess
> > you could block changing the property if VM and thus the backend
> > is not connected and running.
> > 
> > 
> > Which is even more complexity but at least it is consistent.
> 
> Oh, I understand what you thinking about.
> 
> I agreed, but why implementation (in this series) is not consistent? I was
> thinking that current (in current qemu version) impl is not, because
> GET_VRING_BASE has two behaviors, depending on the presence of the protocol
> feature (drain or not drain requests).
> 
> Now, I think it would be more clear: one message for drain GET_VRING_BASE
> (as it was before), one for not drain GET_VRING_BASE_SKIP_DRAIN.


Well the message is consistent. But its use does not seem to be.

> > 
> > 
> > 
> > 
> > 
> > 
> > > I think backend must perform
> > > drain in all cases except live migration with all the necessary checks
> > > (protocol features, special message GET_VRING_BASE_SKIP_DRAIN).
> > 
> > This is not what is going on now, right? Whether it drains depends
> > on protocol features not on VM state and I think we should
> > keep it like this.
> 
> Well yes, now in current qemu version drain or not drain depends on protocol
> feature only, that try to be negotiated only if inflight_migration is set
> on.
> 
> In patch: in case live migration (if protocol feature and  device param)
> sent message GET_VRING_BASE_SKIP_DRAIN to skip drain, in other cases sent
> other message - GET_VRING_BASE with draining all requests.
> So drain or not drain depends on message that QEMU send to backend.
> 
> > 
> > 
> > > 
> > >              i turn inflight off.
> > >              now it looks like it will happily migrate?
> > > 
> > >          So in this case, when vm migrated with inflight off, it will leads to using
> > >          GET_VRING_BASE message, that wait all requests to be drained.
> > > 
> > > 
> > >                           --
> > >                           2.34.1
> > > 
> > > 
> > 
> > i mean it migrated with on. vm was stopped then it flipped to off.
> After stop vm will send GET_VRING_BASE to backend, that lead to drain all
> requests. Non of the inflight requests will be missed here, If that's what
> you mean.

this:

+    bool skip_drain = vhost_user_blk_inflight_needed(s) &&
+                      runstate_check(RUN_STATE_FINISH_MIGRATE);

is what I dislike.

I dislike poking at migration state. It's painful enough that we
need to worry about VM running or not.

> > 
> > 
> > 
> > 
> > 

Re: [PATCH v5 4/6] vhost-user-blk: make inflight-migration prop mutable

[Alexandr Moshkov]

On 7/29/26 19:48, Michael S. Tsirkin wrote:
> On Wed, Jul 29, 2026 at 07:07:11PM +0500, Alexandr Moshkov wrote:
>> On 7/29/26 17:13, Michael S. Tsirkin wrote:
>>> On Wed, Jul 29, 2026 at 03:56:54PM +0500, Alexandr Moshkov wrote:
>>>> On 7/29/26 15:48, Michael S. Tsirkin wrote:
>>>>
>>>>       On Wed, Jul 29, 2026 at 03:35:31PM +0500, Alexandr Moshkov wrote:
>>>>
>>>>           On 7/29/26 15:14, Michael S. Tsirkin wrote:
>>>>
>>>>               On Wed, Jul 29, 2026 at 03:09:57PM +0500, Alexandr Moshkov wrote:
>>>>
>>>>                   On 7/29/26 14:32, Michael S. Tsirkin wrote:
>>>>
>>>>                        On Tue, Jul 28, 2026 at 03:08:39PM +0500, Alexandr Moshkov wrote:
>>>>
>>>>                            When migrating from a QEMU version that supports inflight-migration to
>>>>                            an older one that does not, there is no way to disable the feature at
>>>>                            runtime — the VM must be stopped and reconfigured. This is impractical
>>>>                            in production environments.
>>>>
>>>>                            Make the inflight-migration property mutable after device realization
>>>>                            so it can be toggled via qom-set without restarting the VM.
>>>>
>>>>                            Acked-by: Raphael Norwitz <rnorwitz@nvidia.com>
>>>>                            Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
>>>>
>>>>
>>>>
>>>>                        I think I am beginning to understand.
>>>>
>>>>
>>>>                        You are running qemu with inflight-migration on and want to migrate
>>>>                        to qemu without inflight-migration at all.
>>>>
>>>>
>>>>                        Since it is guest transparent you could retrofit it like this.
>>>>
>>>>                        My question is why is it worth it, we do not normally support
>>>>                        migrating between qemu versions with different command lines.
>>>>
>>>>                   I think you right. Maybe I've been focusing too much on the ability to migrate
>>>>                   between versions of qemu with and without inflight migration.
>>>>
>>>>                   This series allows to turn off and on the inflight-migration feature at runtime
>>>>                   without recreating the VM
>>> restarting you mean.
>> Yes, sorry.
>>>> (it was the only way to turn off the feature, since
>>>>                   the protocol feature could no longer be turned off, after initialization with
>>>>                   the backend). This is more important feature, and it leads to the fact that
>>>>                   this feature allows to migrate between different versions of qemu (with and
>>>>                   without inflight-migration support).
>>> so again it's not the 1st feature we have like this. we tie migration
>>> to the machine type and same set of command line flags
>>> specifically to keep things manageable.
>>>
>>> really cross version migration is a pain as it is.
>> I understand that, thanks! Maybe we should focus on the fact that this
>> series allows to turn inflight_migration on and off (without cross version
>> migration) ?
>>
>> Firstly, I wanted to implement this through migration capabilities, but in
>> one of the threads,
>> it was agreed that a separate capability only for vhost-user-blk is not a
>> good idea.
> I mean it's a generic thing.
>
>> Therefore, I implemented it through a parameter that can be changed using
>> qom-set.
>>
>>>>                            ---
>>>>                             hw/block/vhost-user-blk.c | 10 ++++++++--
>>>>                             1 file changed, 8 insertions(+), 2 deletions(-)
>>>>
>>>>                            diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
>>>>                            index e3b873af7c..650c004bde 100644
>>>>                            --- a/hw/block/vhost-user-blk.c
>>>>                            +++ b/hw/block/vhost-user-blk.c
>>>>                            @@ -619,6 +619,8 @@ static const VMStateDescription vmstate_vhost_user_blk = {
>>>>                                 }
>>>>                             };
>>>>
>>>>                            +static PropertyInfo vhost_user_blk_inflight_migration_prop;
>>>>                            +
>>>>                             static const Property vhost_user_blk_properties[] = {
>>>>                                 DEFINE_PROP_CHR("chardev", VHostUserBlk, chardev),
>>>>                                 DEFINE_PROP_UINT16("num-queues", VHostUserBlk, num_queues,
>>>>                            @@ -632,8 +634,9 @@ static const Property vhost_user_blk_properties[] = {
>>>>                                                   VIRTIO_BLK_F_WRITE_ZEROES, true),
>>>>                                 DEFINE_PROP_BOOL("skip-get-vring-base-on-force-shutdown", VHostUserBlk,
>>>>                                                  skip_get_vring_base_on_force_shutdown, false),
>>>>                            -    DEFINE_PROP_BOOL("inflight-migration", VHostUserBlk,
>>>>                            -                     inflight_migration, false),
>>>>                            +    DEFINE_PROP("inflight-migration", VHostUserBlk, inflight_migration,
>>>>                            +                vhost_user_blk_inflight_migration_prop, bool,
>>>>                            +                .set_default = true, .defval.u = false),
>>>>                             };
>>>>
>>>>                             static void vhost_user_blk_class_init(ObjectClass *klass, const void *data)
>>>>                            @@ -665,6 +668,9 @@ static const TypeInfo vhost_user_blk_info = {
>>>>
>>>>                             static void virtio_register_types(void)
>>>>                             {
>>>>                            +    vhost_user_blk_inflight_migration_prop = qdev_prop_bool;
>>>>                            +    vhost_user_blk_inflight_migration_prop.realized_set_allowed = true;
>>>>                            +
>>>>                                 type_register_static(&vhost_user_blk_info);
>>>>                             }
>>>>
>>>>                        So then,  for example, let us say I paused the VM, then set the flag,
>>>>                        now inflight is on but GET_BASE did not drain it?
>>>>
>>>>                   If I understood the question correctly, this is valid behavior. Before
>>>>                   migration QEMU check protocol features to understand does the backend support
>>>>                   inflight migration. If it does, after that QEMU migrate inflight buffer to
>>>>                   other VM. If it's not, return error before migration started.
>>>>
>>>>               I apologise i reverted the logic in the question.
>>>>
>>>>
>>>>               I start vm and inflight is on.
>>>>               I stop vm get base does not drain.
>>>>
>>>>           In case of VM stop, backend wait to drain all requests. Ability to perform
>>>>           drain or not available only during migration by skip_drain variable in
>>>>           vhost_user_blk_stop().
>>>>
>>>>       What if it was migration but it failed?
>>>>
>>>> If migration failed, source vm just keep using inflight region, backend
>>>> continue to execute inflight requests that was tried to migrate.
>>> Sorry, to be more clear. I mean migration succeeded but destination failed to
>>> start, so now vm is stopped and we are now trying migrating to a different destination.
>> Is this a real case? I was thinking in case of any destination errors
>> (migration process or starting), source vm has to work...
> It does not start automatically in all cases, no.
>
>> I guess in that case, inflight requests can only be executed on different
>> destination or in current destination if vm be able to start.
>>
>>> IIUC currently on VM stop we simply send GET_BASE and this stops
>>> backend, and depending on features things remain in the inflight buffer.
>>> Correct me if  I am wrong.
>> Yes backend has to disconnect from QEMU.. So it not be changing guest
>> memory.
>>> I think that we do not want a slow flush on vm stop if we
>>> can avoid it, and we also do not want backend to keep changing
>>> guest memory when VM is stopped. No?
>> Yes, it is
>>>>           skip_drain true only if inflight_migration is on and runstate is
>>>>           FINISH_MIGRATION.
>>> FINISH_MIGRATE? actually i do not see where it affects it.
>> Sorry, I didn't understand) Who affects who?
>
>>>>       This btw I don't much like, a stopped VM would preferably
>>>>       behave the same whatever the reason to stop.
>>>>
>>>> Well, I don't see any other way to implement this.
>>> I am not 100% sure it's implementable as described, but I guess
>>> you could block changing the property if VM and thus the backend
>>> is not connected and running.
>>>
>>>
>>> Which is even more complexity but at least it is consistent.
>> Oh, I understand what you thinking about.
>>
>> I agreed, but why implementation (in this series) is not consistent? I was
>> thinking that current (in current qemu version) impl is not, because
>> GET_VRING_BASE has two behaviors, depending on the presence of the protocol
>> feature (drain or not drain requests).
>>
>> Now, I think it would be more clear: one message for drain GET_VRING_BASE
>> (as it was before), one for not drain GET_VRING_BASE_SKIP_DRAIN.
>
> Well the message is consistent. But its use does not seem to be.
>
>>>
>>>
>>>
>>>
>>>
>>>> I think backend must perform
>>>> drain in all cases except live migration with all the necessary checks
>>>> (protocol features, special message GET_VRING_BASE_SKIP_DRAIN).
>>> This is not what is going on now, right? Whether it drains depends
>>> on protocol features not on VM state and I think we should
>>> keep it like this.
>> Well yes, now in current qemu version drain or not drain depends on protocol
>> feature only, that try to be negotiated only if inflight_migration is set
>> on.
>>
>> In patch: in case live migration (if protocol feature and  device param)
>> sent message GET_VRING_BASE_SKIP_DRAIN to skip drain, in other cases sent
>> other message - GET_VRING_BASE with draining all requests.
>> So drain or not drain depends on message that QEMU send to backend.
>>
>>>
>>>>               i turn inflight off.
>>>>               now it looks like it will happily migrate?
>>>>
>>>>           So in this case, when vm migrated with inflight off, it will leads to using
>>>>           GET_VRING_BASE message, that wait all requests to be drained.
>>>>
>>>>
>>>>                            --
>>>>                            2.34.1
>>>>
>>>>
>>> i mean it migrated with on. vm was stopped then it flipped to off.
>> After stop vm will send GET_VRING_BASE to backend, that lead to drain all
>> requests. Non of the inflight requests will be missed here, If that's what
>> you mean.
> this:
>
> +    bool skip_drain = vhost_user_blk_inflight_needed(s) &&
> +                      runstate_check(RUN_STATE_FINISH_MIGRATE);
>
> is what I dislike.
>
> I dislike poking at migration state. It's painful enough that we
> need to worry about VM running or not.

I understand that. But I don't know how to do it another way. We can't 
just erase runstate check - it will possibly leads to missing inflight 
requests during VM stop.

And don't know how to make stop more consistent. Do you have any thoughts?

>>>
>>>
>>>
>>>

[PATCH v5 5/6] vhost-user-blk: move inflight_needed higher

[Alexandr Moshkov]

Move vhost_user_blk_inflight_needed() earlier in the file so it can
be called from vhost_user_blk_stop(), which is defined before the
VMState section. No functional change.

Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
---
 hw/block/vhost-user-blk.c | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
index 650c004bde..156ed6de30 100644
--- a/hw/block/vhost-user-blk.c
+++ b/hw/block/vhost-user-blk.c
@@ -131,6 +131,17 @@ const VhostDevConfigOps blk_ops = {
     .vhost_dev_config_notifier = vhost_user_blk_handle_config_change,
 };
 
+static bool vhost_user_blk_inflight_needed(void *opaque)
+{
+    struct VHostUserBlk *s = opaque;
+
+    bool inflight_migration = virtio_has_feature(s->dev.protocol_features,
+                               VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
+
+    return inflight_migration;
+}
+
+
 static int vhost_user_blk_start(VirtIODevice *vdev, Error **errp)
 {
     VHostUserBlk *s = VHOST_USER_BLK(vdev);
@@ -587,14 +598,6 @@ static struct vhost_dev *vhost_user_blk_get_vhost(VirtIODevice *vdev)
     return &s->dev;
 }
 
-static bool vhost_user_blk_inflight_needed(void *opaque)
-{
-    struct VHostUserBlk *s = opaque;
-
-    return vhost_user_has_protocol_feature(
-        &s->dev, VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
-}
-
 static const VMStateDescription vmstate_vhost_user_blk_inflight = {
     .name = "vhost-user-blk/inflight",
     .version_id = 1,
-- 
2.34.1

[PATCH v5 6/6] vhost-user-blk: use GET_VRING_BASE_SKIP_DRAIN during inflight migration

[Alexandr Moshkov]

Currently during live migration vhost-user-blk sends GET_VRING_BASE to
stop vrings, which causes the back-end to drain all in-flight I/O before
returning. This blocks the migration source until all I/O completes,
adding significant downtime proportional to the I/O load.

When inflight-migration is enabled and the device is stopping due to
live migration (RUN_STATE_FINISH_MIGRATE), send GET_VRING_BASE_SKIP_DRAIN
instead. This instructs the back-end to immediately suspend in-flight
I/O and record it in the shared inflight region, which is then migrated
to the destination host along with the rest of the device state.

If the back-end does not support VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN,
migration is aborted with an error in pre_save rather than hitting an
assert at runtime.

Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
---
 hw/block/vhost-user-blk.c      | 30 ++++++++++++++++++++++++------
 hw/virtio/vhost-user.c         |  3 +--
 include/hw/virtio/vhost-user.h |  1 -
 3 files changed, 25 insertions(+), 9 deletions(-)

diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
index 156ed6de30..9531d92e48 100644
--- a/hw/block/vhost-user-blk.c
+++ b/hw/block/vhost-user-blk.c
@@ -135,10 +135,7 @@ static bool vhost_user_blk_inflight_needed(void *opaque)
 {
     struct VHostUserBlk *s = opaque;
 
-    bool inflight_migration = virtio_has_feature(s->dev.protocol_features,
-                               VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
-
-    return inflight_migration;
+    return s->inflight_migration;
 }
 
 
@@ -239,11 +236,14 @@ static int vhost_user_blk_stop(VirtIODevice *vdev)
         return 0;
     }
 
+    bool skip_drain = vhost_user_blk_inflight_needed(s) &&
+                      runstate_check(RUN_STATE_FINISH_MIGRATE);
+
     force_stop = s->skip_get_vring_base_on_force_shutdown &&
                  qemu_force_shutdown_requested();
 
     ret = force_stop ? vhost_dev_force_stop(&s->dev, vdev, true) :
-                       vhost_dev_stop(&s->dev, vdev, true, false);
+                       vhost_dev_stop(&s->dev, vdev, true, skip_drain);
 
     err = k->set_guest_notifiers(qbus->parent, s->dev.nvqs, false);
     if (err < 0) {
@@ -375,7 +375,6 @@ static int vhost_user_blk_connect(DeviceState *dev, Error **errp)
     vhost_dev_set_config_notifier(&s->dev, &blk_ops);
 
     s->vhost_user.supports_config = true;
-    s->vhost_user.supports_inflight_migration = s->inflight_migration;
     ret = vhost_dev_init(&s->dev, &s->vhost_user, VHOST_BACKEND_TYPE_USER, 0,
                          errp);
     if (ret < 0) {
@@ -598,10 +597,29 @@ static struct vhost_dev *vhost_user_blk_get_vhost(VirtIODevice *vdev)
     return &s->dev;
 }
 
+static bool vhost_user_blk_pre_save(void *opaque, Error **errp)
+{
+    VHostUserBlk *s = VHOST_USER_BLK(opaque);
+
+    bool inflight_migration_enabled = vhost_user_has_protocol_feature(&s->dev,
+                               VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN);
+
+    if (vhost_user_blk_inflight_needed(s) && !inflight_migration_enabled) {
+        error_setg(errp, "can't migrate vhost-user-blk device: "
+                         "backend doesn't support "
+                         "VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN "
+                         "protocol feature");
+        return false;
+    }
+
+    return true;
+}
+
 static const VMStateDescription vmstate_vhost_user_blk_inflight = {
     .name = "vhost-user-blk/inflight",
     .version_id = 1,
     .needed = vhost_user_blk_inflight_needed,
+    .pre_save_errp = vhost_user_blk_pre_save,
     .fields = (const VMStateField[]) {
         VMSTATE_VHOST_INFLIGHT_REGION(inflight, VHostUserBlk),
         VMSTATE_END_OF_LIST()
diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c
index bf455daa44..03b8bc20f9 100644
--- a/hw/virtio/vhost-user.c
+++ b/hw/virtio/vhost-user.c
@@ -2571,8 +2571,7 @@ static int vhost_user_backend_init(struct vhost_dev *dev, void *opaque,
             }
         }
 
-        if (!u->user->supports_inflight_migration ||
-            !virtio_has_feature(protocol_features,
+        if (!virtio_has_feature(protocol_features,
                                 VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD)) {
             protocol_features &= ~(1ULL <<
                                VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
diff --git a/include/hw/virtio/vhost-user.h b/include/hw/virtio/vhost-user.h
index 47c13f8677..78b4fba27f 100644
--- a/include/hw/virtio/vhost-user.h
+++ b/include/hw/virtio/vhost-user.h
@@ -73,7 +73,6 @@ typedef struct VhostUserState {
     GPtrArray *notifiers;
     int memory_slots;
     bool supports_config;
-    bool supports_inflight_migration;
 } VhostUserState;
 
 /**
-- 
2.34.1

Re: [PATCH v5 0/6] vhost-user-blk: add compatibility with older qemu versions

[Michael S. Tsirkin]

On Tue, Jul 28, 2026 at 03:08:35PM +0500, Alexandr Moshkov wrote:
> v4 -> v5:
> - introduce protocol feature VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN
>   to guard the new message, instead of reusing the existing
>   VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INLIFGHT. Update docs.
> - improved commit messages.
> 
> v3 -> v4:
> - add new protocol message GET_VRING_BASE_SKIP_DRAIN instead of parameter to GET_VRING_BASE message. This new message can be send instead of GET_VRING_BASE, allowing back-end to suspend inflight I/O immediately instead of waiting and completing them.
> - rebase to newer master
> 
> v2 -> v3:
> - fix complile problems
> - add assert check in do_vhost_virtio_stop
> - make inflight-migration property mutable
> 
> v1 -> v2:
> - reorganize commits: make refactor commits first, then core semantic change
> - add additional pre_save check for inflight migration possibility
> 
> ---
> 
> This is v5 of the series previously sent as
> "[PATCH v4 0/6] vhost-user-blk: fix inflight migration compatibility".
> 
> This series extends the vhost-user-blk inflight migration feature
> introduced in QEMU 11.0 to address a runtime compatibility problem.
> 
> Currently, the inflight migration behaviour in vhost-user-blk is
> controlled by VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT, which is
> negotiated once at connection time. Once the feature is negotiated, the
> back-end always uses suspend semantics on GET_VRING_BASE — there is no
> way for the front-end to request normal drain behaviour on a per-call
> basis without reconnecting. This makes it impossible to disable
> inflight-migration at runtime, which is needed when, for example,
> migrating a VM to an older QEMU version that does not support the
> feature.


This part, I do not understand.

What exactly does "migrating" mean here? Live migration?
An older QEMU will presumably either negotiate or fail to negotiate
the bit.


> To solve this, the series introduces a new protocol message
> VHOST_USER_GET_VRING_BASE_SKIP_DRAIN (id=45) guarded by a new protocol
> feature VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN. The message is
> semantically identical to GET_VRING_BASE but explicitly instructs the
> back-end to suspend in-flight I/O immediately rather than draining it.
> This gives the front-end explicit per-call control: GET_VRING_BASE for
> normal operation, GET_VRING_BASE_SKIP_DRAIN during live migration.
> 
> In vhost-user-blk, GET_VRING_BASE_SKIP_DRAIN is sent only when
> inflight-migration is enabled and the device is stopping due to live
> migration (RUN_STATE_FINISH_MIGRATE). The inflight-migration property
> is also made mutable so it can be toggled via qom-set at runtime without
> reconnecting to the back-end.
> 
> Alexandr Moshkov (6):
>   vhost-user: add skip_drain param to do_vhost_virtqueue_stop
>   vhost-user: add GET_VRING_BASE_SKIP_DRAIN message
>   vhost-user: use skip_drain with GET_VRING_BASE_SKIP_DRAIN message
>   vhost-user-blk: make inflight-migration prop mutable
>   vhost-user-blk: move inflight_needed higher
>   vhost-user-blk: use GET_VRING_BASE_SKIP_DRAIN during inflight
>     migration
> 
>  backends/cryptodev-vhost.c        |  2 +-
>  backends/vhost-user.c             |  2 +-
>  docs/interop/vhost-user.rst       | 88 +++++++++++++++++++------------
>  hw/block/vhost-user-blk.c         | 43 ++++++++++++---
>  hw/net/vhost_net.c                |  9 ++--
>  hw/scsi/vhost-scsi-common.c       |  2 +-
>  hw/virtio/vdpa-dev.c              |  2 +-
>  hw/virtio/vhost-user-base.c       |  2 +-
>  hw/virtio/vhost-user-fs.c         |  2 +-
>  hw/virtio/vhost-user-scmi.c       |  2 +-
>  hw/virtio/vhost-user.c            | 47 ++++++++++++++---
>  hw/virtio/vhost-vsock-common.c    |  2 +-
>  hw/virtio/vhost.c                 | 34 ++++++++----
>  include/hw/virtio/vhost-backend.h |  1 +
>  include/hw/virtio/vhost-user.h    |  2 +-
>  include/hw/virtio/vhost.h         |  7 ++-
>  16 files changed, 172 insertions(+), 75 deletions(-)
> 
> -- 
> 2.34.1