[PATCH v3 0/5] vhost-user-blk: fix compatibility with older qemu versions

[PATCH v3 0/5] vhost-user-blk: fix compatibility with older qemu versions

[Alexandr Moshkov]

v2 -> v3:
- fix complile problems
- add assert check in do_vhost_virtio_stop
- make inflight-migration property mutable

v1 -> v2:
- reorganize commits: make refactor commits first, then core semantic change
- add additional pre_save check for inflight migration possibility

---

This is a small continuation of my series about inflight migration for vhost-user-blk.

This series is designed to solve the problem of compatibility with older versions of qemu, where this feature has not yet been introduced (for example, if we want to downgrade versions due to some problems).

In the current version for vhost-user-blk, this feature is enabled using the parameter and further migration of the inflight region will depend on whether the vhost-user has accepted the new protocol feature or not. This creates an inconvenient dependency, because there is no way to disable this feature without reconnecting to the backend.
This series slightly changes the semantics of the introduced protocol feature VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT. Enabling this feature adds a new parameter for GET_VRING_BASE message - skip_drain, which allows to control drain in-flight requests on the backend.

Thus, user can enable or disable inflight-migration param for vhost-user-blk to maintain compatibility with older versions of QEMU.

Alexandr Moshkov (5):
  vhost-user.rst: fix typo
  vhost-user-blk: make inflight-migration prop mutable
  vhost-user: add skip_drain param to do_vhost_virtqueue_stop
  vhost-user-blk: move inflight_needed higher
  vhost-user: add skip_drain param to GET_VRING_BASE

 backends/cryptodev-vhost.c     |  2 +-
 backends/vhost-user.c          |  2 +-
 docs/interop/vhost-user.rst    |  8 +++----
 hw/block/vhost-user-blk.c      | 43 +++++++++++++++++++++++++++-------
 hw/net/vhost_net.c             |  9 +++----
 hw/scsi/vhost-scsi-common.c    |  2 +-
 hw/virtio/vdpa-dev.c           |  2 +-
 hw/virtio/vhost-user-base.c    |  2 +-
 hw/virtio/vhost-user-fs.c      |  2 +-
 hw/virtio/vhost-user-scmi.c    |  2 +-
 hw/virtio/vhost-user.c         |  3 +--
 hw/virtio/vhost-vsock-common.c |  2 +-
 hw/virtio/vhost.c              | 29 ++++++++++++++++-------
 include/hw/virtio/vhost-user.h |  1 -
 include/hw/virtio/vhost.h      |  7 ++++--
 15 files changed, 76 insertions(+), 40 deletions(-)

-- 
2.34.1

[PATCH v3 1/5] vhost-user.rst: fix typo

[Alexandr Moshkov]

Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
---
 docs/interop/vhost-user.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
index 137c9f3669..bfa75ff9a3 100644
--- a/docs/interop/vhost-user.rst
+++ b/docs/interop/vhost-user.rst
@@ -1268,7 +1268,7 @@ Front-end message types
   How to suspend an in-flight request depends on the implementation of the back-end
   but it typically can be done by aborting or cancelling the underlying I/O
   request. The ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT``
-  protocol feature must only be neogotiated if
+  protocol feature must only be negotiated if
   ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD`` is also negotiated.
 
 ``VHOST_USER_SET_VRING_KICK``
-- 
2.34.1

[PATCH v3 2/5] vhost-user-blk: make inflight-migration prop mutable

[Alexandr Moshkov]

Now this property can be turned on or off using the qom-set

Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
---
 hw/block/vhost-user-blk.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
index c151e83677..179f027cb4 100644
--- a/hw/block/vhost-user-blk.c
+++ b/hw/block/vhost-user-blk.c
@@ -603,6 +603,8 @@ static const VMStateDescription vmstate_vhost_user_blk = {
     }
 };
 
+static PropertyInfo vhost_user_blk_inflight_migration_prop;
+
 static const Property vhost_user_blk_properties[] = {
     DEFINE_PROP_CHR("chardev", VHostUserBlk, chardev),
     DEFINE_PROP_UINT16("num-queues", VHostUserBlk, num_queues,
@@ -616,8 +618,9 @@ static const Property vhost_user_blk_properties[] = {
                       VIRTIO_BLK_F_WRITE_ZEROES, true),
     DEFINE_PROP_BOOL("skip-get-vring-base-on-force-shutdown", VHostUserBlk,
                      skip_get_vring_base_on_force_shutdown, false),
-    DEFINE_PROP_BOOL("inflight-migration", VHostUserBlk,
-                     inflight_migration, false),
+    DEFINE_PROP("inflight-migration", VHostUserBlk, inflight_migration,
+                vhost_user_blk_inflight_migration_prop, bool,
+                .set_default = true, .defval.u = false),
 };
 
 static void vhost_user_blk_class_init(ObjectClass *klass, const void *data)
@@ -649,6 +652,9 @@ static const TypeInfo vhost_user_blk_info = {
 
 static void virtio_register_types(void)
 {
+    vhost_user_blk_inflight_migration_prop = qdev_prop_bool;
+    vhost_user_blk_inflight_migration_prop.realized_set_allowed = true;
+
     type_register_static(&vhost_user_blk_info);
 }
 
-- 
2.34.1

[PATCH v3 3/5] vhost-user: add skip_drain param to do_vhost_virtqueue_stop

[Alexandr Moshkov]

Will be used in future commit for adding this param to GET_VRING_BASE

Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
---
 backends/cryptodev-vhost.c     |  2 +-
 backends/vhost-user.c          |  2 +-
 hw/block/vhost-user-blk.c      |  2 +-
 hw/net/vhost_net.c             |  9 +++++----
 hw/scsi/vhost-scsi-common.c    |  2 +-
 hw/virtio/vdpa-dev.c           |  2 +-
 hw/virtio/vhost-user-base.c    |  2 +-
 hw/virtio/vhost-user-fs.c      |  2 +-
 hw/virtio/vhost-user-scmi.c    |  2 +-
 hw/virtio/vhost-vsock-common.c |  2 +-
 hw/virtio/vhost.c              | 28 +++++++++++++++++++---------
 include/hw/virtio/vhost.h      |  7 +++++--
 12 files changed, 38 insertions(+), 24 deletions(-)

diff --git a/backends/cryptodev-vhost.c b/backends/cryptodev-vhost.c
index 943680a23a..7a457380d0 100644
--- a/backends/cryptodev-vhost.c
+++ b/backends/cryptodev-vhost.c
@@ -110,7 +110,7 @@ static void
 cryptodev_vhost_stop_one(CryptoDevBackendVhost *crypto,
                                  VirtIODevice *dev)
 {
-    vhost_dev_stop(&crypto->dev, dev, false);
+    vhost_dev_stop(&crypto->dev, dev, false, false);
     vhost_dev_disable_notifiers(&crypto->dev, dev);
 }
 
diff --git a/backends/vhost-user.c b/backends/vhost-user.c
index 42845329e7..10be713ebd 100644
--- a/backends/vhost-user.c
+++ b/backends/vhost-user.c
@@ -108,7 +108,7 @@ vhost_user_backend_stop(VhostUserBackend *b)
         return 0;
     }
 
-    ret = vhost_dev_stop(&b->dev, b->vdev, true);
+    ret = vhost_dev_stop(&b->dev, b->vdev, true, false);
 
     if (k->set_guest_notifiers &&
         k->set_guest_notifiers(qbus->parent, b->dev.nvqs, false) < 0) {
diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
index 179f027cb4..718fc94a29 100644
--- a/hw/block/vhost-user-blk.c
+++ b/hw/block/vhost-user-blk.c
@@ -225,7 +225,7 @@ static int vhost_user_blk_stop(VirtIODevice *vdev)
                  qemu_force_shutdown_requested();
 
     ret = force_stop ? vhost_dev_force_stop(&s->dev, vdev, true) :
-                       vhost_dev_stop(&s->dev, vdev, true);
+                       vhost_dev_stop(&s->dev, vdev, true, false);
 
     if (k->set_guest_notifiers(qbus->parent, s->dev.nvqs, false) < 0) {
         error_report("vhost guest notifier cleanup failed: %d", ret);
diff --git a/hw/net/vhost_net.c b/hw/net/vhost_net.c
index a8ee18a912..604f19e03a 100644
--- a/hw/net/vhost_net.c
+++ b/hw/net/vhost_net.c
@@ -384,7 +384,7 @@ fail:
     if (net->nc->info->poll) {
         net->nc->info->poll(net->nc, true);
     }
-    vhost_dev_stop(&net->dev, dev, false);
+    vhost_dev_stop(&net->dev, dev, false, false);
 fail_start:
     return r;
 }
@@ -403,7 +403,7 @@ static void vhost_net_stop_one(struct vhost_net *net,
     if (net->nc->info->poll) {
         net->nc->info->poll(net->nc, true);
     }
-    vhost_dev_stop(&net->dev, dev, false);
+    vhost_dev_stop(&net->dev, dev, false, false);
     if (net->nc->info->stop) {
         net->nc->info->stop(net->nc);
     }
@@ -641,7 +641,8 @@ void vhost_net_virtqueue_reset(VirtIODevice *vdev, NetClientState *nc,
     vhost_virtqueue_stop(&net->dev,
                          vdev,
                          net->dev.vqs + idx,
-                         net->dev.vq_index + idx);
+                         net->dev.vq_index + idx,
+                         false);
 }
 
 int vhost_net_virtqueue_restart(VirtIODevice *vdev, NetClientState *nc,
@@ -691,7 +692,7 @@ err_start:
         assert(ret >= 0);
     }
 
-    vhost_dev_stop(&net->dev, vdev, false);
+    vhost_dev_stop(&net->dev, vdev, false, false);
 
     return r;
 }
diff --git a/hw/scsi/vhost-scsi-common.c b/hw/scsi/vhost-scsi-common.c
index 0bb4305de6..41c1e45aac 100644
--- a/hw/scsi/vhost-scsi-common.c
+++ b/hw/scsi/vhost-scsi-common.c
@@ -108,7 +108,7 @@ int vhost_scsi_common_stop(VHostSCSICommon *vsc)
     VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(qbus);
     int ret = 0;
 
-    ret = vhost_dev_stop(&vsc->dev, vdev, true);
+    ret = vhost_dev_stop(&vsc->dev, vdev, true, false);
 
     if (k->set_guest_notifiers) {
         int r = k->set_guest_notifiers(qbus->parent, vsc->dev.nvqs, false);
diff --git a/hw/virtio/vdpa-dev.c b/hw/virtio/vdpa-dev.c
index 4532d63653..7279f18442 100644
--- a/hw/virtio/vdpa-dev.c
+++ b/hw/virtio/vdpa-dev.c
@@ -301,7 +301,7 @@ static void vhost_vdpa_device_stop(VirtIODevice *vdev)
         return;
     }
 
-    vhost_dev_stop(&s->dev, vdev, false);
+    vhost_dev_stop(&s->dev, vdev, false, false);
 
     ret = k->set_guest_notifiers(qbus->parent, s->dev.nvqs, false);
     if (ret < 0) {
diff --git a/hw/virtio/vhost-user-base.c b/hw/virtio/vhost-user-base.c
index 01ab9ca56b..9d3875a04b 100644
--- a/hw/virtio/vhost-user-base.c
+++ b/hw/virtio/vhost-user-base.c
@@ -77,7 +77,7 @@ static int vub_stop(VirtIODevice *vdev)
         return 0;
     }
 
-    ret = vhost_dev_stop(&vub->vhost_dev, vdev, true);
+    ret = vhost_dev_stop(&vub->vhost_dev, vdev, true, false);
 
     if (k->set_guest_notifiers(qbus->parent, vub->vhost_dev.nvqs, false) < 0) {
         error_report("vhost guest notifier cleanup failed: %d", ret);
diff --git a/hw/virtio/vhost-user-fs.c b/hw/virtio/vhost-user-fs.c
index ad6fcacf06..2f6f6df67f 100644
--- a/hw/virtio/vhost-user-fs.c
+++ b/hw/virtio/vhost-user-fs.c
@@ -111,7 +111,7 @@ static int vuf_stop(VirtIODevice *vdev)
         return 0;
     }
 
-    ret = vhost_dev_stop(&fs->vhost_dev, vdev, true);
+    ret = vhost_dev_stop(&fs->vhost_dev, vdev, true, false);
 
     if (k->set_guest_notifiers(qbus->parent, fs->vhost_dev.nvqs, false) < 0) {
         error_report("vhost guest notifier cleanup failed: %d", ret);
diff --git a/hw/virtio/vhost-user-scmi.c b/hw/virtio/vhost-user-scmi.c
index f9264c4374..dbde342a6e 100644
--- a/hw/virtio/vhost-user-scmi.c
+++ b/hw/virtio/vhost-user-scmi.c
@@ -101,7 +101,7 @@ static int vu_scmi_stop(VirtIODevice *vdev)
         return 0;
     }
 
-    ret = vhost_dev_stop(vhost_dev, vdev, true);
+    ret = vhost_dev_stop(vhost_dev, vdev, true, false);
 
     if (k->set_guest_notifiers(qbus->parent, vhost_dev->nvqs, false) < 0) {
         error_report("vhost guest notifier cleanup failed: %d", ret);
diff --git a/hw/virtio/vhost-vsock-common.c b/hw/virtio/vhost-vsock-common.c
index b33def900a..3d086ed825 100644
--- a/hw/virtio/vhost-vsock-common.c
+++ b/hw/virtio/vhost-vsock-common.c
@@ -106,7 +106,7 @@ int vhost_vsock_common_stop(VirtIODevice *vdev)
         return 0;
     }
 
-    ret = vhost_dev_stop(&vvc->vhost_dev, vdev, true);
+    ret = vhost_dev_stop(&vvc->vhost_dev, vdev, true, false);
 
     if (k->set_guest_notifiers(qbus->parent, vvc->vhost_dev.nvqs, false) < 0) {
         error_report("vhost guest notifier cleanup failed: %d", ret);
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index b4cdb7762f..ac4d6fca73 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -23,6 +23,7 @@
 #include "qemu/log.h"
 #include "standard-headers/linux/vhost_types.h"
 #include "hw/virtio/virtio-bus.h"
+#include "hw/virtio/vhost-user.h"
 #include "hw/mem/memory-device.h"
 #include "migration/blocker.h"
 #include "migration/qemu-file-types.h"
@@ -1387,8 +1388,13 @@ fail_alloc_desc:
 static int do_vhost_virtqueue_stop(struct vhost_dev *dev,
                                    struct VirtIODevice *vdev,
                                    struct vhost_virtqueue *vq,
-                                   unsigned idx, bool force)
+                                   unsigned idx, bool force,
+                                   bool skip_drain)
 {
+    if (skip_drain) {
+        assert(virtio_has_feature(dev->protocol_features,
+                VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT));
+    }
     int vhost_vq_index = dev->vhost_ops->vhost_get_vq_index(dev, idx);
     struct vhost_vring_state state = {
         .index = vhost_vq_index,
@@ -1439,9 +1445,10 @@ static int do_vhost_virtqueue_stop(struct vhost_dev *dev,
 int vhost_virtqueue_stop(struct vhost_dev *dev,
                          struct VirtIODevice *vdev,
                          struct vhost_virtqueue *vq,
-                         unsigned idx)
+                         unsigned idx,
+                         bool skip_drain)
 {
-    return do_vhost_virtqueue_stop(dev, vdev, vq, idx, false);
+    return do_vhost_virtqueue_stop(dev, vdev, vq, idx, false, skip_drain);
 }
 
 static int vhost_virtqueue_set_busyloop_timeout(struct vhost_dev *dev,
@@ -2220,7 +2227,8 @@ fail_vq:
         vhost_virtqueue_stop(hdev,
                              vdev,
                              hdev->vqs + i,
-                             hdev->vq_index + i);
+                             hdev->vq_index + i,
+                             false);
     }
 
 fail_mem:
@@ -2235,7 +2243,7 @@ fail_features:
 
 /* Host notifiers must be enabled at this point. */
 static int do_vhost_dev_stop(struct vhost_dev *hdev, VirtIODevice *vdev,
-                             bool vrings, bool force)
+                             bool vrings, bool force, bool skip_drain)
 {
     int i;
     int rc = 0;
@@ -2262,7 +2270,8 @@ static int do_vhost_dev_stop(struct vhost_dev *hdev, VirtIODevice *vdev,
                                       vdev,
                                       hdev->vqs + i,
                                       hdev->vq_index + i,
-                                      force);
+                                      force,
+                                      skip_drain);
     }
     if (hdev->vhost_ops->vhost_reset_status) {
         hdev->vhost_ops->vhost_reset_status(hdev);
@@ -2282,15 +2291,16 @@ static int do_vhost_dev_stop(struct vhost_dev *hdev, VirtIODevice *vdev,
     return rc;
 }
 
-int vhost_dev_stop(struct vhost_dev *hdev, VirtIODevice *vdev, bool vrings)
+int vhost_dev_stop(struct vhost_dev *hdev, VirtIODevice *vdev, bool vrings,
+                   bool skip_drain)
 {
-    return do_vhost_dev_stop(hdev, vdev, vrings, false);
+    return do_vhost_dev_stop(hdev, vdev, vrings, false, skip_drain);
 }
 
 int vhost_dev_force_stop(struct vhost_dev *hdev, VirtIODevice *vdev,
                          bool vrings)
 {
-    return do_vhost_dev_stop(hdev, vdev, vrings, true);
+    return do_vhost_dev_stop(hdev, vdev, vrings, true, false);
 }
 
 int vhost_net_set_backend(struct vhost_dev *hdev,
diff --git a/include/hw/virtio/vhost.h b/include/hw/virtio/vhost.h
index 89817bd848..3881f6784b 100644
--- a/include/hw/virtio/vhost.h
+++ b/include/hw/virtio/vhost.h
@@ -233,6 +233,7 @@ int vhost_dev_start(struct vhost_dev *hdev, VirtIODevice *vdev, bool vrings);
  * @hdev: common vhost_dev structure
  * @vdev: the VirtIODevice structure
  * @vrings: true to have vrings disabled in this call
+ * @skip_drain: true to notice back-end to skip draining all in-flight requests
  *
  * Stop the vhost device. After the device is stopped the notifiers
  * can be disabled (@vhost_dev_disable_notifiers) and the device can
@@ -240,7 +241,8 @@ int vhost_dev_start(struct vhost_dev *hdev, VirtIODevice *vdev, bool vrings);
  *
  * Return: 0 on success, != 0 on error when stopping dev.
  */
-int vhost_dev_stop(struct vhost_dev *hdev, VirtIODevice *vdev, bool vrings);
+int vhost_dev_stop(struct vhost_dev *hdev, VirtIODevice *vdev, bool vrings,
+                   bool skip_drain);
 
 /**
  * vhost_dev_force_stop() - force stop the vhost device
@@ -398,7 +400,8 @@ int vhost_device_iotlb_miss(struct vhost_dev *dev, uint64_t iova, int write);
 int vhost_virtqueue_start(struct vhost_dev *dev, struct VirtIODevice *vdev,
                           struct vhost_virtqueue *vq, unsigned idx);
 int vhost_virtqueue_stop(struct vhost_dev *dev, struct VirtIODevice *vdev,
-                         struct vhost_virtqueue *vq, unsigned idx);
+                         struct vhost_virtqueue *vq, unsigned idx,
+                         bool skip_drain);
 
 void vhost_dev_reset_inflight(struct vhost_inflight *inflight);
 void vhost_dev_free_inflight(struct vhost_inflight *inflight);
-- 
2.34.1

[PATCH v3 4/5] vhost-user-blk: move inflight_needed higher

[Alexandr Moshkov]

Will be used in future commits for adding skip_drain param

Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
---
 hw/block/vhost-user-blk.c | 21 +++++++++++----------
 1 file changed, 11 insertions(+), 10 deletions(-)

diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
index 718fc94a29..a3ecd83f54 100644
--- a/hw/block/vhost-user-blk.c
+++ b/hw/block/vhost-user-blk.c
@@ -130,6 +130,17 @@ const VhostDevConfigOps blk_ops = {
     .vhost_dev_config_notifier = vhost_user_blk_handle_config_change,
 };
 
+static bool vhost_user_blk_inflight_needed(void *opaque)
+{
+    struct VHostUserBlk *s = opaque;
+
+    bool inflight_migration = virtio_has_feature(s->dev.protocol_features,
+                               VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
+
+    return inflight_migration;
+}
+
+
 static int vhost_user_blk_start(VirtIODevice *vdev, Error **errp)
 {
     VHostUserBlk *s = VHOST_USER_BLK(vdev);
@@ -569,16 +580,6 @@ static struct vhost_dev *vhost_user_blk_get_vhost(VirtIODevice *vdev)
     return &s->dev;
 }
 
-static bool vhost_user_blk_inflight_needed(void *opaque)
-{
-    struct VHostUserBlk *s = opaque;
-
-    bool inflight_migration = virtio_has_feature(s->dev.protocol_features,
-                               VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
-
-    return inflight_migration;
-}
-
 static const VMStateDescription vmstate_vhost_user_blk_inflight = {
     .name = "vhost-user-blk/inflight",
     .version_id = 1,
-- 
2.34.1

[PATCH v3 5/5] vhost-user: add skip_drain param to GET_VRING_BASE

[Alexandr Moshkov]

In case of migration of QEMU from the new version (where the
inllight-migration parameter is present), to the old one (where it is
absent) there is no way to disable this feature on the backend during
runtime.

This commit slightly changes the semantics of the protocol feature
VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT. Enabling this feature
adds a new parameter for GET_VRING_BASE, which allows to control the
drain in-flight requests on the backend.
Thus, QEMU will be able to turn this feature on GET_VRING_BASE off and
on anytime.

In vhost-user-blk use inflight_migration param to enable skip_drain to
suspend in-flight I/O requests, and then migrate them throught inflight
subsection.

Also now QEMU will always try to setup
VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT protocol featrue with
backend. This will allow to use skip_drain parameter on GET_VRING_BASE
message.

Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
---
 docs/interop/vhost-user.rst    |  6 ++----
 hw/block/vhost-user-blk.c      | 30 ++++++++++++++++++++++++------
 hw/virtio/vhost-user.c         |  3 +--
 hw/virtio/vhost.c              |  1 +
 include/hw/virtio/vhost-user.h |  1 -
 5 files changed, 28 insertions(+), 13 deletions(-)

diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
index bfa75ff9a3..63efc87264 100644
--- a/docs/interop/vhost-user.rst
+++ b/docs/interop/vhost-user.rst
@@ -1255,14 +1255,12 @@ Front-end message types
   *suspended*, see :ref:`Suspended device state
   <suspended_device_state>`.
 
-  The request payload's *num* field is currently reserved and must be
-  set to 0.
-
   By default, the back-end must complete all inflight I/O requests for the
   specified vring before stopping it.
 
   If the ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT`` protocol
-  feature has been negotiated, the back-end may suspend in-flight I/O
+  feature has been negotiated, using request payload's *num* field,
+  when *num* is set to 1, QEMU can tell the back-end to suspend in-flight I/O
   requests and record them as described in :ref:`Inflight I/O tracking
   <inflight_io_tracking>` instead of completing them before stopping the vring.
   How to suspend an in-flight request depends on the implementation of the back-end
diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
index a3ecd83f54..522e9d34b5 100644
--- a/hw/block/vhost-user-blk.c
+++ b/hw/block/vhost-user-blk.c
@@ -134,10 +134,7 @@ static bool vhost_user_blk_inflight_needed(void *opaque)
 {
     struct VHostUserBlk *s = opaque;
 
-    bool inflight_migration = virtio_has_feature(s->dev.protocol_features,
-                               VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
-
-    return inflight_migration;
+    return s->inflight_migration;
 }
 
 
@@ -232,11 +229,14 @@ static int vhost_user_blk_stop(VirtIODevice *vdev)
         return 0;
     }
 
+    bool skip_drain = vhost_user_blk_inflight_needed(s) &&
+                      runstate_check(RUN_STATE_FINISH_MIGRATE);
+
     force_stop = s->skip_get_vring_base_on_force_shutdown &&
                  qemu_force_shutdown_requested();
 
     ret = force_stop ? vhost_dev_force_stop(&s->dev, vdev, true) :
-                       vhost_dev_stop(&s->dev, vdev, true, false);
+                       vhost_dev_stop(&s->dev, vdev, true, skip_drain);
 
     if (k->set_guest_notifiers(qbus->parent, s->dev.nvqs, false) < 0) {
         error_report("vhost guest notifier cleanup failed: %d", ret);
@@ -364,7 +364,6 @@ static int vhost_user_blk_connect(DeviceState *dev, Error **errp)
     vhost_dev_set_config_notifier(&s->dev, &blk_ops);
 
     s->vhost_user.supports_config = true;
-    s->vhost_user.supports_inflight_migration = s->inflight_migration;
     ret = vhost_dev_init(&s->dev, &s->vhost_user, VHOST_BACKEND_TYPE_USER, 0,
                          errp);
     if (ret < 0) {
@@ -580,10 +579,29 @@ static struct vhost_dev *vhost_user_blk_get_vhost(VirtIODevice *vdev)
     return &s->dev;
 }
 
+static bool vhost_user_blk_pre_save(void *opaque, Error **errp)
+{
+    VHostUserBlk *s = VHOST_USER_BLK(opaque);
+
+    bool inflight_migration_enabled = virtio_has_feature(
+        s->dev.protocol_features,
+        VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
+    if (vhost_user_blk_inflight_needed(s) && !inflight_migration_enabled) {
+        error_setg(errp, "can't migrate vhost-user-blk device: "
+                         "backend doesn't support "
+                         "VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT "
+                         "protocol feature");
+        return false;
+    }
+
+    return true;
+}
+
 static const VMStateDescription vmstate_vhost_user_blk_inflight = {
     .name = "vhost-user-blk/inflight",
     .version_id = 1,
     .needed = vhost_user_blk_inflight_needed,
+    .pre_save_errp = vhost_user_blk_pre_save,
     .fields = (const VMStateField[]) {
         VMSTATE_VHOST_INFLIGHT_REGION(inflight, VHostUserBlk),
         VMSTATE_END_OF_LIST()
diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c
index bb8f8eab77..ed95ec7523 100644
--- a/hw/virtio/vhost-user.c
+++ b/hw/virtio/vhost-user.c
@@ -2225,8 +2225,7 @@ static int vhost_user_backend_init(struct vhost_dev *dev, void *opaque,
             }
         }
 
-        if (!u->user->supports_inflight_migration ||
-            !virtio_has_feature(protocol_features,
+        if (!virtio_has_feature(protocol_features,
                                 VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD)) {
             protocol_features &= ~(1ULL <<
                                VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index ac4d6fca73..54e4ce2660 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -1398,6 +1398,7 @@ static int do_vhost_virtqueue_stop(struct vhost_dev *dev,
     int vhost_vq_index = dev->vhost_ops->vhost_get_vq_index(dev, idx);
     struct vhost_vring_state state = {
         .index = vhost_vq_index,
+        .num = skip_drain,
     };
     int r = 0;
 
diff --git a/include/hw/virtio/vhost-user.h b/include/hw/virtio/vhost-user.h
index 53fe996686..c95bad5ddc 100644
--- a/include/hw/virtio/vhost-user.h
+++ b/include/hw/virtio/vhost-user.h
@@ -69,7 +69,6 @@ typedef struct VhostUserState {
     GPtrArray *notifiers;
     int memory_slots;
     bool supports_config;
-    bool supports_inflight_migration;
 } VhostUserState;
 
 /**
-- 
2.34.1

Re: [PATCH v3 0/5] vhost-user-blk: fix compatibility with older qemu versions

[Alexandr Moshkov]

Ping :)

On 3/30/26 14:52, Alexandr Moshkov wrote:
> v2 -> v3:
> - fix complile problems
> - add assert check in do_vhost_virtio_stop
> - make inflight-migration property mutable
>
> v1 -> v2:
> - reorganize commits: make refactor commits first, then core semantic change
> - add additional pre_save check for inflight migration possibility
>
> ---
>
> This is a small continuation of my series about inflight migration for vhost-user-blk.
>
> This series is designed to solve the problem of compatibility with older versions of qemu, where this feature has not yet been introduced (for example, if we want to downgrade versions due to some problems).
>
> In the current version for vhost-user-blk, this feature is enabled using the parameter and further migration of the inflight region will depend on whether the vhost-user has accepted the new protocol feature or not. This creates an inconvenient dependency, because there is no way to disable this feature without reconnecting to the backend.
> This series slightly changes the semantics of the introduced protocol feature VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT. Enabling this feature adds a new parameter for GET_VRING_BASE message - skip_drain, which allows to control drain in-flight requests on the backend.
>
> Thus, user can enable or disable inflight-migration param for vhost-user-blk to maintain compatibility with older versions of QEMU.
>
> Alexandr Moshkov (5):
>    vhost-user.rst: fix typo
>    vhost-user-blk: make inflight-migration prop mutable
>    vhost-user: add skip_drain param to do_vhost_virtqueue_stop
>    vhost-user-blk: move inflight_needed higher
>    vhost-user: add skip_drain param to GET_VRING_BASE
>
>   backends/cryptodev-vhost.c     |  2 +-
>   backends/vhost-user.c          |  2 +-
>   docs/interop/vhost-user.rst    |  8 +++----
>   hw/block/vhost-user-blk.c      | 43 +++++++++++++++++++++++++++-------
>   hw/net/vhost_net.c             |  9 +++----
>   hw/scsi/vhost-scsi-common.c    |  2 +-
>   hw/virtio/vdpa-dev.c           |  2 +-
>   hw/virtio/vhost-user-base.c    |  2 +-
>   hw/virtio/vhost-user-fs.c      |  2 +-
>   hw/virtio/vhost-user-scmi.c    |  2 +-
>   hw/virtio/vhost-user.c         |  3 +--
>   hw/virtio/vhost-vsock-common.c |  2 +-
>   hw/virtio/vhost.c              | 29 ++++++++++++++++-------
>   include/hw/virtio/vhost-user.h |  1 -
>   include/hw/virtio/vhost.h      |  7 ++++--
>   15 files changed, 76 insertions(+), 40 deletions(-)
>

Re: [PATCH v3 0/5] vhost-user-blk: fix compatibility with older qemu versions

[Alexandr Moshkov]

Greetings! Gentle ping

On 4/6/26 14:51, Alexandr Moshkov wrote:
> Ping :)
>
> On 3/30/26 14:52, Alexandr Moshkov wrote:
>> v2 -> v3:
>> - fix complile problems
>> - add assert check in do_vhost_virtio_stop
>> - make inflight-migration property mutable
>>
>> v1 -> v2:
>> - reorganize commits: make refactor commits first, then core semantic 
>> change
>> - add additional pre_save check for inflight migration possibility
>>
>> ---
>>
>> This is a small continuation of my series about inflight migration 
>> for vhost-user-blk.
>>
>> This series is designed to solve the problem of compatibility with 
>> older versions of qemu, where this feature has not yet been 
>> introduced (for example, if we want to downgrade versions due to some 
>> problems).
>>
>> In the current version for vhost-user-blk, this feature is enabled 
>> using the parameter and further migration of the inflight region will 
>> depend on whether the vhost-user has accepted the new protocol 
>> feature or not. This creates an inconvenient dependency, because 
>> there is no way to disable this feature without reconnecting to the 
>> backend.
>> This series slightly changes the semantics of the introduced protocol 
>> feature VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT. Enabling this 
>> feature adds a new parameter for GET_VRING_BASE message - skip_drain, 
>> which allows to control drain in-flight requests on the backend.
>>
>> Thus, user can enable or disable inflight-migration param for 
>> vhost-user-blk to maintain compatibility with older versions of QEMU.
>>
>> Alexandr Moshkov (5):
>>    vhost-user.rst: fix typo
>>    vhost-user-blk: make inflight-migration prop mutable
>>    vhost-user: add skip_drain param to do_vhost_virtqueue_stop
>>    vhost-user-blk: move inflight_needed higher
>>    vhost-user: add skip_drain param to GET_VRING_BASE
>>
>>   backends/cryptodev-vhost.c     |  2 +-
>>   backends/vhost-user.c          |  2 +-
>>   docs/interop/vhost-user.rst    |  8 +++----
>>   hw/block/vhost-user-blk.c      | 43 +++++++++++++++++++++++++++-------
>>   hw/net/vhost_net.c             |  9 +++----
>>   hw/scsi/vhost-scsi-common.c    |  2 +-
>>   hw/virtio/vdpa-dev.c           |  2 +-
>>   hw/virtio/vhost-user-base.c    |  2 +-
>>   hw/virtio/vhost-user-fs.c      |  2 +-
>>   hw/virtio/vhost-user-scmi.c    |  2 +-
>>   hw/virtio/vhost-user.c         |  3 +--
>>   hw/virtio/vhost-vsock-common.c |  2 +-
>>   hw/virtio/vhost.c              | 29 ++++++++++++++++-------
>>   include/hw/virtio/vhost-user.h |  1 -
>>   include/hw/virtio/vhost.h      |  7 ++++--
>>   15 files changed, 76 insertions(+), 40 deletions(-)
>>

Re: [PATCH v3 5/5] vhost-user: add skip_drain param to GET_VRING_BASE

[Raphael Norwitz]

I’m not sure I like using the num field in vhost_vring_state to set 
magic values which affect protocol behavior for GET_VRING_BASE. It feels 
like a hack to me. I would think the proper solution if we want to 
support migration from new to old would be to have new use a different 
new message entirely. Can we do that?

On 3/30/26 11:52 AM, Alexandr Moshkov wrote:
> In case of migration of QEMU from the new version (where the
> inllight-migration parameter is present), to the old one (where it is

nit: spelling inllight-migration

> absent) there is no way to disable this feature on the backend during
> runtime.
> 
> This commit slightly changes the semantics of the protocol feature
> VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT. Enabling this feature
> adds a new parameter for GET_VRING_BASE, which allows to control the
> drain in-flight requests on the backend.
> Thus, QEMU will be able to turn this feature on GET_VRING_BASE off and
> on anytime.
> 
> In vhost-user-blk use inflight_migration param to enable skip_drain to
> suspend in-flight I/O requests, and then migrate them throught inflight
> subsection.
> 
> Also now QEMU will always try to setup
> VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT protocol featrue with
> backend. This will allow to use skip_drain parameter on GET_VRING_BASE
> message.
> 
> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
> Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
> ---
>   docs/interop/vhost-user.rst    |  6 ++----
>   hw/block/vhost-user-blk.c      | 30 ++++++++++++++++++++++++------
>   hw/virtio/vhost-user.c         |  3 +--
>   hw/virtio/vhost.c              |  1 +
>   include/hw/virtio/vhost-user.h |  1 -
>   5 files changed, 28 insertions(+), 13 deletions(-)
> 
> diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
> index bfa75ff9a3..63efc87264 100644
> --- a/docs/interop/vhost-user.rst
> +++ b/docs/interop/vhost-user.rst
> @@ -1255,14 +1255,12 @@ Front-end message types
>     *suspended*, see :ref:`Suspended device state
>     <suspended_device_state>`.
>   
> -  The request payload's *num* field is currently reserved and must be
> -  set to 0.
> -
>     By default, the back-end must complete all inflight I/O requests for the
>     specified vring before stopping it.
>   
>     If the ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT`` protocol
> -  feature has been negotiated, the back-end may suspend in-flight I/O
> +  feature has been negotiated, using request payload's *num* field,
> +  when *num* is set to 1, QEMU can tell the back-end to suspend in-flight I/O
>     requests and record them as described in :ref:`Inflight I/O tracking
>     <inflight_io_tracking>` instead of completing them before stopping the vring.
>     How to suspend an in-flight request depends on the implementation of the back-end
> diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
> index a3ecd83f54..522e9d34b5 100644
> --- a/hw/block/vhost-user-blk.c
> +++ b/hw/block/vhost-user-blk.c
> @@ -134,10 +134,7 @@ static bool vhost_user_blk_inflight_needed(void *opaque)
>   {
>       struct VHostUserBlk *s = opaque;
>   
> -    bool inflight_migration = virtio_has_feature(s->dev.protocol_features,
> -                               VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
> -
> -    return inflight_migration;
> +    return s->inflight_migration;
>   }
>   
>   
> @@ -232,11 +229,14 @@ static int vhost_user_blk_stop(VirtIODevice *vdev)
>           return 0;
>       }
>   
> +    bool skip_drain = vhost_user_blk_inflight_needed(s) &&

nit: declarations on top

> +                      runstate_check(RUN_STATE_FINISH_MIGRATE);
> +
>       force_stop = s->skip_get_vring_base_on_force_shutdown &&
>                    qemu_force_shutdown_requested();
>   
>       ret = force_stop ? vhost_dev_force_stop(&s->dev, vdev, true) :
> -                       vhost_dev_stop(&s->dev, vdev, true, false);
> +                       vhost_dev_stop(&s->dev, vdev, true, skip_drain);
>   
>       if (k->set_guest_notifiers(qbus->parent, s->dev.nvqs, false) < 0) {
>           error_report("vhost guest notifier cleanup failed: %d", ret);
> @@ -364,7 +364,6 @@ static int vhost_user_blk_connect(DeviceState *dev, Error **errp)
>       vhost_dev_set_config_notifier(&s->dev, &blk_ops);
>   
>       s->vhost_user.supports_config = true;
> -    s->vhost_user.supports_inflight_migration = s->inflight_migration;
>       ret = vhost_dev_init(&s->dev, &s->vhost_user, VHOST_BACKEND_TYPE_USER, 0,
>                            errp);
>       if (ret < 0) {
> @@ -580,10 +579,29 @@ static struct vhost_dev *vhost_user_blk_get_vhost(VirtIODevice *vdev)
>       return &s->dev;
>   }
>   
> +static bool vhost_user_blk_pre_save(void *opaque, Error **errp)
> +{
> +    VHostUserBlk *s = VHOST_USER_BLK(opaque);
> +
> +    bool inflight_migration_enabled = virtio_has_feature(
> +        s->dev.protocol_features,
> +        VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
> +    if (vhost_user_blk_inflight_needed(s) && !inflight_migration_enabled) {
> +        error_setg(errp, "can't migrate vhost-user-blk device: "
> +                         "backend doesn't support "
> +                         "VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT "
> +                         "protocol feature");
> +        return false;
> +    }
> +
> +    return true;
> +}
> +
>   static const VMStateDescription vmstate_vhost_user_blk_inflight = {
>       .name = "vhost-user-blk/inflight",
>       .version_id = 1,
>       .needed = vhost_user_blk_inflight_needed,
> +    .pre_save_errp = vhost_user_blk_pre_save,
>       .fields = (const VMStateField[]) {
>           VMSTATE_VHOST_INFLIGHT_REGION(inflight, VHostUserBlk),
>           VMSTATE_END_OF_LIST()
> diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c
> index bb8f8eab77..ed95ec7523 100644
> --- a/hw/virtio/vhost-user.c
> +++ b/hw/virtio/vhost-user.c
> @@ -2225,8 +2225,7 @@ static int vhost_user_backend_init(struct vhost_dev *dev, void *opaque,
>               }
>           }
>   
> -        if (!u->user->supports_inflight_migration ||
> -            !virtio_has_feature(protocol_features,
> +        if (!virtio_has_feature(protocol_features,
>                                   VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD)) {
>               protocol_features &= ~(1ULL <<
>                                  VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
> diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
> index ac4d6fca73..54e4ce2660 100644
> --- a/hw/virtio/vhost.c
> +++ b/hw/virtio/vhost.c
> @@ -1398,6 +1398,7 @@ static int do_vhost_virtqueue_stop(struct vhost_dev *dev,
>       int vhost_vq_index = dev->vhost_ops->vhost_get_vq_index(dev, idx);
>       struct vhost_vring_state state = {
>           .index = vhost_vq_index,
> +        .num = skip_drain,
>       };
>       int r = 0;
>   
> diff --git a/include/hw/virtio/vhost-user.h b/include/hw/virtio/vhost-user.h
> index 53fe996686..c95bad5ddc 100644
> --- a/include/hw/virtio/vhost-user.h
> +++ b/include/hw/virtio/vhost-user.h
> @@ -69,7 +69,6 @@ typedef struct VhostUserState {
>       GPtrArray *notifiers;
>       int memory_slots;
>       bool supports_config;
> -    bool supports_inflight_migration;
>   } VhostUserState;
>   
>   /**

Re: [PATCH v3 1/5] vhost-user.rst: fix typo

[Raphael Norwitz]

Reviewed-by: Raphael Norwitz <rnorwitz@nvidia.com>

On 3/30/26 11:52 AM, Alexandr Moshkov wrote:
> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
> ---
>   docs/interop/vhost-user.rst | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
> index 137c9f3669..bfa75ff9a3 100644
> --- a/docs/interop/vhost-user.rst
> +++ b/docs/interop/vhost-user.rst
> @@ -1268,7 +1268,7 @@ Front-end message types
>     How to suspend an in-flight request depends on the implementation of the back-end
>     but it typically can be done by aborting or cancelling the underlying I/O
>     request. The ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT``
> -  protocol feature must only be neogotiated if
> +  protocol feature must only be negotiated if
>     ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD`` is also negotiated.
>   
>   ``VHOST_USER_SET_VRING_KICK``

Re: [PATCH v3 2/5] vhost-user-blk: make inflight-migration prop mutable

[Raphael Norwitz]

Acked-by: Raphael Norwitz <rnorwitz@nvidia.com>

On 3/30/26 11:52 AM, Alexandr Moshkov wrote:
> Now this property can be turned on or off using the qom-set
> 
> Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
> ---
>   hw/block/vhost-user-blk.c | 10 ++++++++--
>   1 file changed, 8 insertions(+), 2 deletions(-)
> 
> diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
> index c151e83677..179f027cb4 100644
> --- a/hw/block/vhost-user-blk.c
> +++ b/hw/block/vhost-user-blk.c
> @@ -603,6 +603,8 @@ static const VMStateDescription vmstate_vhost_user_blk = {
>       }
>   };
>   
> +static PropertyInfo vhost_user_blk_inflight_migration_prop;
> +
>   static const Property vhost_user_blk_properties[] = {
>       DEFINE_PROP_CHR("chardev", VHostUserBlk, chardev),
>       DEFINE_PROP_UINT16("num-queues", VHostUserBlk, num_queues,
> @@ -616,8 +618,9 @@ static const Property vhost_user_blk_properties[] = {
>                         VIRTIO_BLK_F_WRITE_ZEROES, true),
>       DEFINE_PROP_BOOL("skip-get-vring-base-on-force-shutdown", VHostUserBlk,
>                        skip_get_vring_base_on_force_shutdown, false),
> -    DEFINE_PROP_BOOL("inflight-migration", VHostUserBlk,
> -                     inflight_migration, false),
> +    DEFINE_PROP("inflight-migration", VHostUserBlk, inflight_migration,
> +                vhost_user_blk_inflight_migration_prop, bool,
> +                .set_default = true, .defval.u = false),
>   };
>   
>   static void vhost_user_blk_class_init(ObjectClass *klass, const void *data)
> @@ -649,6 +652,9 @@ static const TypeInfo vhost_user_blk_info = {
>   
>   static void virtio_register_types(void)
>   {
> +    vhost_user_blk_inflight_migration_prop = qdev_prop_bool;
> +    vhost_user_blk_inflight_migration_prop.realized_set_allowed = true;
> +
>       type_register_static(&vhost_user_blk_info);
>   }
>   

Re: [PATCH v3 5/5] vhost-user: add skip_drain param to GET_VRING_BASE

[Alexandr Moshkov]

[-- Attachment #1: Type: text/plain, Size: 8446 bytes --]

Greetings! Thanks for reply!


On 4/15/26 20:21, Raphael Norwitz wrote:
> I’m not sure I like using the num field in vhost_vring_state to set 
> magic values which affect protocol behavior for GET_VRING_BASE. It 
> feels like a hack to me. I would think the proper solution if we want 
> to support migration from new to old would be to have new use a 
> different new message entirely. Can we do that?

I think we can, but I thought at first that this will be almost a 
complete copy of GET_VRING_BASE message, with the exception of waiting 
for drain of requests, so I choose to expand existing message.

If this is not an appropriate approach, is it better to make a new 
message like GET_VRING_BASE or a separate message used together with 
default GET_VRING_BASE message (for example, message for setting some 
kind of status on the server)? What should I name this new message?

I also fix other comments, thanks!

>
> On 3/30/26 11:52 AM, Alexandr Moshkov wrote:
>> In case of migration of QEMU from the new version (where the
>> inllight-migration parameter is present), to the old one (where it is
>
> nit: spelling inllight-migration
>
>> absent) there is no way to disable this feature on the backend during
>> runtime.
>>
>> This commit slightly changes the semantics of the protocol feature
>> VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT. Enabling this feature
>> adds a new parameter for GET_VRING_BASE, which allows to control the
>> drain in-flight requests on the backend.
>> Thus, QEMU will be able to turn this feature on GET_VRING_BASE off and
>> on anytime.
>>
>> In vhost-user-blk use inflight_migration param to enable skip_drain to
>> suspend in-flight I/O requests, and then migrate them throught inflight
>> subsection.
>>
>> Also now QEMU will always try to setup
>> VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT protocol featrue with
>> backend. This will allow to use skip_drain parameter on GET_VRING_BASE
>> message.
>>
>> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
>> Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
>> ---
>>   docs/interop/vhost-user.rst    |  6 ++----
>>   hw/block/vhost-user-blk.c      | 30 ++++++++++++++++++++++++------
>>   hw/virtio/vhost-user.c         |  3 +--
>>   hw/virtio/vhost.c              |  1 +
>>   include/hw/virtio/vhost-user.h |  1 -
>>   5 files changed, 28 insertions(+), 13 deletions(-)
>>
>> diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
>> index bfa75ff9a3..63efc87264 100644
>> --- a/docs/interop/vhost-user.rst
>> +++ b/docs/interop/vhost-user.rst
>> @@ -1255,14 +1255,12 @@ Front-end message types
>>     *suspended*, see :ref:`Suspended device state
>>     <suspended_device_state>`.
>>   -  The request payload's *num* field is currently reserved and must be
>> -  set to 0.
>> -
>>     By default, the back-end must complete all inflight I/O requests 
>> for the
>>     specified vring before stopping it.
>>       If the ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT`` protocol
>> -  feature has been negotiated, the back-end may suspend in-flight I/O
>> +  feature has been negotiated, using request payload's *num* field,
>> +  when *num* is set to 1, QEMU can tell the back-end to suspend 
>> in-flight I/O
>>     requests and record them as described in :ref:`Inflight I/O tracking
>>     <inflight_io_tracking>` instead of completing them before 
>> stopping the vring.
>>     How to suspend an in-flight request depends on the implementation 
>> of the back-end
>> diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
>> index a3ecd83f54..522e9d34b5 100644
>> --- a/hw/block/vhost-user-blk.c
>> +++ b/hw/block/vhost-user-blk.c
>> @@ -134,10 +134,7 @@ static bool vhost_user_blk_inflight_needed(void 
>> *opaque)
>>   {
>>       struct VHostUserBlk *s = opaque;
>>   -    bool inflight_migration = 
>> virtio_has_feature(s->dev.protocol_features,
>> - VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
>> -
>> -    return inflight_migration;
>> +    return s->inflight_migration;
>>   }
>>     @@ -232,11 +229,14 @@ static int vhost_user_blk_stop(VirtIODevice 
>> *vdev)
>>           return 0;
>>       }
>>   +    bool skip_drain = vhost_user_blk_inflight_needed(s) &&
>
> nit: declarations on top
>
>> + runstate_check(RUN_STATE_FINISH_MIGRATE);
>> +
>>       force_stop = s->skip_get_vring_base_on_force_shutdown &&
>>                    qemu_force_shutdown_requested();
>>         ret = force_stop ? vhost_dev_force_stop(&s->dev, vdev, true) :
>> -                       vhost_dev_stop(&s->dev, vdev, true, false);
>> +                       vhost_dev_stop(&s->dev, vdev, true, skip_drain);
>>         if (k->set_guest_notifiers(qbus->parent, s->dev.nvqs, false) 
>> < 0) {
>>           error_report("vhost guest notifier cleanup failed: %d", ret);
>> @@ -364,7 +364,6 @@ static int vhost_user_blk_connect(DeviceState 
>> *dev, Error **errp)
>>       vhost_dev_set_config_notifier(&s->dev, &blk_ops);
>>         s->vhost_user.supports_config = true;
>> -    s->vhost_user.supports_inflight_migration = s->inflight_migration;
>>       ret = vhost_dev_init(&s->dev, &s->vhost_user, 
>> VHOST_BACKEND_TYPE_USER, 0,
>>                            errp);
>>       if (ret < 0) {
>> @@ -580,10 +579,29 @@ static struct vhost_dev 
>> *vhost_user_blk_get_vhost(VirtIODevice *vdev)
>>       return &s->dev;
>>   }
>>   +static bool vhost_user_blk_pre_save(void *opaque, Error **errp)
>> +{
>> +    VHostUserBlk *s = VHOST_USER_BLK(opaque);
>> +
>> +    bool inflight_migration_enabled = virtio_has_feature(
>> +        s->dev.protocol_features,
>> +        VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
>> +    if (vhost_user_blk_inflight_needed(s) && 
>> !inflight_migration_enabled) {
>> +        error_setg(errp, "can't migrate vhost-user-blk device: "
>> +                         "backend doesn't support "
>> + "VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT "
>> +                         "protocol feature");
>> +        return false;
>> +    }
>> +
>> +    return true;
>> +}
>> +
>>   static const VMStateDescription vmstate_vhost_user_blk_inflight = {
>>       .name = "vhost-user-blk/inflight",
>>       .version_id = 1,
>>       .needed = vhost_user_blk_inflight_needed,
>> +    .pre_save_errp = vhost_user_blk_pre_save,
>>       .fields = (const VMStateField[]) {
>>           VMSTATE_VHOST_INFLIGHT_REGION(inflight, VHostUserBlk),
>>           VMSTATE_END_OF_LIST()
>> diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c
>> index bb8f8eab77..ed95ec7523 100644
>> --- a/hw/virtio/vhost-user.c
>> +++ b/hw/virtio/vhost-user.c
>> @@ -2225,8 +2225,7 @@ static int vhost_user_backend_init(struct 
>> vhost_dev *dev, void *opaque,
>>               }
>>           }
>>   -        if (!u->user->supports_inflight_migration ||
>> -            !virtio_has_feature(protocol_features,
>> +        if (!virtio_has_feature(protocol_features,
>> VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD)) {
>>               protocol_features &= ~(1ULL <<
>> VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
>> diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
>> index ac4d6fca73..54e4ce2660 100644
>> --- a/hw/virtio/vhost.c
>> +++ b/hw/virtio/vhost.c
>> @@ -1398,6 +1398,7 @@ static int do_vhost_virtqueue_stop(struct 
>> vhost_dev *dev,
>>       int vhost_vq_index = dev->vhost_ops->vhost_get_vq_index(dev, idx);
>>       struct vhost_vring_state state = {
>>           .index = vhost_vq_index,
>> +        .num = skip_drain,
>>       };
>>       int r = 0;
>>   diff --git a/include/hw/virtio/vhost-user.h 
>> b/include/hw/virtio/vhost-user.h
>> index 53fe996686..c95bad5ddc 100644
>> --- a/include/hw/virtio/vhost-user.h
>> +++ b/include/hw/virtio/vhost-user.h
>> @@ -69,7 +69,6 @@ typedef struct VhostUserState {
>>       GPtrArray *notifiers;
>>       int memory_slots;
>>       bool supports_config;
>> -    bool supports_inflight_migration;
>>   } VhostUserState;
>>     /**
>

[-- Attachment #2: Type: text/html, Size: 13183 bytes --]

Re: [PATCH v3 0/5] vhost-user-blk: fix compatibility with older qemu versions

[Alexandr Moshkov]

Hello! Another gentle ping :)

On 4/6/26 14:51, Alexandr Moshkov wrote:
> Ping :)
>
> On 3/30/26 14:52, Alexandr Moshkov wrote:
>> v2 -> v3:
>> - fix complile problems
>> - add assert check in do_vhost_virtio_stop
>> - make inflight-migration property mutable
>>
>> v1 -> v2:
>> - reorganize commits: make refactor commits first, then core semantic 
>> change
>> - add additional pre_save check for inflight migration possibility
>>
>> ---
>>
>> This is a small continuation of my series about inflight migration 
>> for vhost-user-blk.
>>
>> This series is designed to solve the problem of compatibility with 
>> older versions of qemu, where this feature has not yet been 
>> introduced (for example, if we want to downgrade versions due to some 
>> problems).
>>
>> In the current version for vhost-user-blk, this feature is enabled 
>> using the parameter and further migration of the inflight region will 
>> depend on whether the vhost-user has accepted the new protocol 
>> feature or not. This creates an inconvenient dependency, because 
>> there is no way to disable this feature without reconnecting to the 
>> backend.
>> This series slightly changes the semantics of the introduced protocol 
>> feature VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT. Enabling this 
>> feature adds a new parameter for GET_VRING_BASE message - skip_drain, 
>> which allows to control drain in-flight requests on the backend.
>>
>> Thus, user can enable or disable inflight-migration param for 
>> vhost-user-blk to maintain compatibility with older versions of QEMU.
>>
>> Alexandr Moshkov (5):
>>    vhost-user.rst: fix typo
>>    vhost-user-blk: make inflight-migration prop mutable
>>    vhost-user: add skip_drain param to do_vhost_virtqueue_stop
>>    vhost-user-blk: move inflight_needed higher
>>    vhost-user: add skip_drain param to GET_VRING_BASE
>>
>>   backends/cryptodev-vhost.c     |  2 +-
>>   backends/vhost-user.c          |  2 +-
>>   docs/interop/vhost-user.rst    |  8 +++----
>>   hw/block/vhost-user-blk.c      | 43 +++++++++++++++++++++++++++-------
>>   hw/net/vhost_net.c             |  9 +++----
>>   hw/scsi/vhost-scsi-common.c    |  2 +-
>>   hw/virtio/vdpa-dev.c           |  2 +-
>>   hw/virtio/vhost-user-base.c    |  2 +-
>>   hw/virtio/vhost-user-fs.c      |  2 +-
>>   hw/virtio/vhost-user-scmi.c    |  2 +-
>>   hw/virtio/vhost-user.c         |  3 +--
>>   hw/virtio/vhost-vsock-common.c |  2 +-
>>   hw/virtio/vhost.c              | 29 ++++++++++++++++-------
>>   include/hw/virtio/vhost-user.h |  1 -
>>   include/hw/virtio/vhost.h      |  7 ++++--
>>   15 files changed, 76 insertions(+), 40 deletions(-)
>>

Re: [PATCH v3 5/5] vhost-user: add skip_drain param to GET_VRING_BASE

[Raphael Norwitz]

On 4/16/26 5:26 AM, Alexandr Moshkov wrote:
> Greetings! Thanks for reply!
> 
> 
> On 4/15/26 20:21, Raphael Norwitz wrote:
>> I’m not sure I like using the num field in vhost_vring_state to set 
>> magic values which affect protocol behavior for GET_VRING_BASE. It 
>> feels like a hack to me. I would think the proper solution if we want 
>> to support migration from new to old would be to have new use a 
>> different new message entirely. Can we do that?
> 
> I think we can, but I thought at first that this will be almost a 
> complete copy of GET_VRING_BASE message, with the exception of waiting 
> for drain of requests, so I choose to expand existing message.
> 

I think a new message would be cleaner. Anyone else have thoughts?

> If this is not an appropriate approach, is it better to make a new 
> message like GET_VRING_BASE or a separate message used together with 
> default GET_VRING_BASE message (for example, message for setting some 
> kind of status on the server)? What should I name this new message?
> 

Maybe GET_VRING_BASE_SKIP_DRAIN? How would a separate message used 
together with default GET_VRING_BASE message work?

> I also fix other comments, thanks!
> 
>>
>> On 3/30/26 11:52 AM, Alexandr Moshkov wrote:
>>> In case of migration of QEMU from the new version (where the
>>> inllight-migration parameter is present), to the old one (where it is
>>
>> nit: spelling inllight-migration
>>
>>> absent) there is no way to disable this feature on the backend during
>>> runtime.
>>>
>>> This commit slightly changes the semantics of the protocol feature
>>> VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT. Enabling this feature
>>> adds a new parameter for GET_VRING_BASE, which allows to control the
>>> drain in-flight requests on the backend.
>>> Thus, QEMU will be able to turn this feature on GET_VRING_BASE off and
>>> on anytime.
>>>
>>> In vhost-user-blk use inflight_migration param to enable skip_drain to
>>> suspend in-flight I/O requests, and then migrate them throught inflight
>>> subsection.
>>>
>>> Also now QEMU will always try to setup
>>> VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT protocol featrue with
>>> backend. This will allow to use skip_drain parameter on GET_VRING_BASE
>>> message.
>>>
>>> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
>>> Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
>>> ---
>>>   docs/interop/vhost-user.rst    |  6 ++----
>>>   hw/block/vhost-user-blk.c      | 30 ++++++++++++++++++++++++------
>>>   hw/virtio/vhost-user.c         |  3 +--
>>>   hw/virtio/vhost.c              |  1 +
>>>   include/hw/virtio/vhost-user.h |  1 -
>>>   5 files changed, 28 insertions(+), 13 deletions(-)
>>>
>>> diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
>>> index bfa75ff9a3..63efc87264 100644
>>> --- a/docs/interop/vhost-user.rst
>>> +++ b/docs/interop/vhost-user.rst
>>> @@ -1255,14 +1255,12 @@ Front-end message types
>>>     *suspended*, see :ref:`Suspended device state
>>>     <suspended_device_state>`.
>>>   -  The request payload's *num* field is currently reserved and must be
>>> -  set to 0.
>>> -
>>>     By default, the back-end must complete all inflight I/O requests 
>>> for the
>>>     specified vring before stopping it.
>>>       If the ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT`` protocol
>>> -  feature has been negotiated, the back-end may suspend in-flight I/O
>>> +  feature has been negotiated, using request payload's *num* field,
>>> +  when *num* is set to 1, QEMU can tell the back-end to suspend in- 
>>> flight I/O
>>>     requests and record them as described in :ref:`Inflight I/O tracking
>>>     <inflight_io_tracking>` instead of completing them before 
>>> stopping the vring.
>>>     How to suspend an in-flight request depends on the implementation 
>>> of the back-end
>>> diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
>>> index a3ecd83f54..522e9d34b5 100644
>>> --- a/hw/block/vhost-user-blk.c
>>> +++ b/hw/block/vhost-user-blk.c
>>> @@ -134,10 +134,7 @@ static bool vhost_user_blk_inflight_needed(void 
>>> *opaque)
>>>   {
>>>       struct VHostUserBlk *s = opaque;
>>>   -    bool inflight_migration = virtio_has_feature(s- 
>>> >dev.protocol_features,
>>> - VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
>>> -
>>> -    return inflight_migration;
>>> +    return s->inflight_migration;
>>>   }
>>>     @@ -232,11 +229,14 @@ static int vhost_user_blk_stop(VirtIODevice 
>>> *vdev)
>>>           return 0;
>>>       }
>>>   +    bool skip_drain = vhost_user_blk_inflight_needed(s) &&
>>
>> nit: declarations on top
>>
>>> + runstate_check(RUN_STATE_FINISH_MIGRATE);
>>> +
>>>       force_stop = s->skip_get_vring_base_on_force_shutdown &&
>>>                    qemu_force_shutdown_requested();
>>>         ret = force_stop ? vhost_dev_force_stop(&s->dev, vdev, true) :
>>> -                       vhost_dev_stop(&s->dev, vdev, true, false);
>>> +                       vhost_dev_stop(&s->dev, vdev, true, skip_drain);
>>>         if (k->set_guest_notifiers(qbus->parent, s->dev.nvqs, false) 
>>> < 0) {
>>>           error_report("vhost guest notifier cleanup failed: %d", ret);
>>> @@ -364,7 +364,6 @@ static int vhost_user_blk_connect(DeviceState 
>>> *dev, Error **errp)
>>>       vhost_dev_set_config_notifier(&s->dev, &blk_ops);
>>>         s->vhost_user.supports_config = true;
>>> -    s->vhost_user.supports_inflight_migration = s->inflight_migration;
>>>       ret = vhost_dev_init(&s->dev, &s->vhost_user, 
>>> VHOST_BACKEND_TYPE_USER, 0,
>>>                            errp);
>>>       if (ret < 0) {
>>> @@ -580,10 +579,29 @@ static struct vhost_dev 
>>> *vhost_user_blk_get_vhost(VirtIODevice *vdev)
>>>       return &s->dev;
>>>   }
>>>   +static bool vhost_user_blk_pre_save(void *opaque, Error **errp)
>>> +{
>>> +    VHostUserBlk *s = VHOST_USER_BLK(opaque);
>>> +
>>> +    bool inflight_migration_enabled = virtio_has_feature(
>>> +        s->dev.protocol_features,
>>> +        VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
>>> +    if (vhost_user_blk_inflight_needed(s) && ! 
>>> inflight_migration_enabled) {
>>> +        error_setg(errp, "can't migrate vhost-user-blk device: "
>>> +                         "backend doesn't support "
>>> + "VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT "
>>> +                         "protocol feature");
>>> +        return false;
>>> +    }
>>> +
>>> +    return true;
>>> +}
>>> +
>>>   static const VMStateDescription vmstate_vhost_user_blk_inflight = {
>>>       .name = "vhost-user-blk/inflight",
>>>       .version_id = 1,
>>>       .needed = vhost_user_blk_inflight_needed,
>>> +    .pre_save_errp = vhost_user_blk_pre_save,
>>>       .fields = (const VMStateField[]) {
>>>           VMSTATE_VHOST_INFLIGHT_REGION(inflight, VHostUserBlk),
>>>           VMSTATE_END_OF_LIST()
>>> diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c
>>> index bb8f8eab77..ed95ec7523 100644
>>> --- a/hw/virtio/vhost-user.c
>>> +++ b/hw/virtio/vhost-user.c
>>> @@ -2225,8 +2225,7 @@ static int vhost_user_backend_init(struct 
>>> vhost_dev *dev, void *opaque,
>>>               }
>>>           }
>>>   -        if (!u->user->supports_inflight_migration ||
>>> -            !virtio_has_feature(protocol_features,
>>> +        if (!virtio_has_feature(protocol_features,
>>> VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD)) {
>>>               protocol_features &= ~(1ULL <<
>>> VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
>>> diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
>>> index ac4d6fca73..54e4ce2660 100644
>>> --- a/hw/virtio/vhost.c
>>> +++ b/hw/virtio/vhost.c
>>> @@ -1398,6 +1398,7 @@ static int do_vhost_virtqueue_stop(struct 
>>> vhost_dev *dev,
>>>       int vhost_vq_index = dev->vhost_ops->vhost_get_vq_index(dev, idx);
>>>       struct vhost_vring_state state = {
>>>           .index = vhost_vq_index,
>>> +        .num = skip_drain,
>>>       };
>>>       int r = 0;
>>>   diff --git a/include/hw/virtio/vhost-user.h b/include/hw/virtio/ 
>>> vhost-user.h
>>> index 53fe996686..c95bad5ddc 100644
>>> --- a/include/hw/virtio/vhost-user.h
>>> +++ b/include/hw/virtio/vhost-user.h
>>> @@ -69,7 +69,6 @@ typedef struct VhostUserState {
>>>       GPtrArray *notifiers;
>>>       int memory_slots;
>>>       bool supports_config;
>>> -    bool supports_inflight_migration;
>>>   } VhostUserState;
>>>     /**
>>

Re: [PATCH v3 5/5] vhost-user: add skip_drain param to GET_VRING_BASE

[Alexandr Moshkov]

Thanks for reply!

On 4/23/26 19:15, Raphael Norwitz wrote:
>
>
> On 4/16/26 5:26 AM, Alexandr Moshkov wrote:
>> Greetings! Thanks for reply!
>>
>>
>> On 4/15/26 20:21, Raphael Norwitz wrote:
>>> I’m not sure I like using the num field in vhost_vring_state to set 
>>> magic values which affect protocol behavior for GET_VRING_BASE. It 
>>> feels like a hack to me. I would think the proper solution if we 
>>> want to support migration from new to old would be to have new use a 
>>> different new message entirely. Can we do that?
>>
>> I think we can, but I thought at first that this will be almost a 
>> complete copy of GET_VRING_BASE message, with the exception of 
>> waiting for drain of requests, so I choose to expand existing message.
>>
>
> I think a new message would be cleaner. Anyone else have thoughts?
>
>> If this is not an appropriate approach, is it better to make a new 
>> message like GET_VRING_BASE or a separate message used together with 
>> default GET_VRING_BASE message (for example, message for setting some 
>> kind of status on the server)? What should I name this new message?
>>
>
> Maybe GET_VRING_BASE_SKIP_DRAIN? How would a separate message used 
> together with default GET_VRING_BASE message work? 

I was thinking about a message something like SET_VRING_ENABLE - for 
example SET_SKIP_DRAIN_ENABLE, that would enable/disable some state 
(skip_drain) in the backend.

>
>> I also fix other comments, thanks!
>>
>>>
>>> On 3/30/26 11:52 AM, Alexandr Moshkov wrote:
>>>> In case of migration of QEMU from the new version (where the
>>>> inllight-migration parameter is present), to the old one (where it is
>>>
>>> nit: spelling inllight-migration
>>>
>>>> absent) there is no way to disable this feature on the backend during
>>>> runtime.
>>>>
>>>> This commit slightly changes the semantics of the protocol feature
>>>> VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT. Enabling this feature
>>>> adds a new parameter for GET_VRING_BASE, which allows to control the
>>>> drain in-flight requests on the backend.
>>>> Thus, QEMU will be able to turn this feature on GET_VRING_BASE off and
>>>> on anytime.
>>>>
>>>> In vhost-user-blk use inflight_migration param to enable skip_drain to
>>>> suspend in-flight I/O requests, and then migrate them throught 
>>>> inflight
>>>> subsection.
>>>>
>>>> Also now QEMU will always try to setup
>>>> VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT protocol featrue with
>>>> backend. This will allow to use skip_drain parameter on GET_VRING_BASE
>>>> message.
>>>>
>>>> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
>>>> Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
>>>> ---
>>>>   docs/interop/vhost-user.rst    |  6 ++----
>>>>   hw/block/vhost-user-blk.c      | 30 ++++++++++++++++++++++++------
>>>>   hw/virtio/vhost-user.c         |  3 +--
>>>>   hw/virtio/vhost.c              |  1 +
>>>>   include/hw/virtio/vhost-user.h |  1 -
>>>>   5 files changed, 28 insertions(+), 13 deletions(-)
>>>>
>>>> diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
>>>> index bfa75ff9a3..63efc87264 100644
>>>> --- a/docs/interop/vhost-user.rst
>>>> +++ b/docs/interop/vhost-user.rst
>>>> @@ -1255,14 +1255,12 @@ Front-end message types
>>>>     *suspended*, see :ref:`Suspended device state
>>>>     <suspended_device_state>`.
>>>>   -  The request payload's *num* field is currently reserved and 
>>>> must be
>>>> -  set to 0.
>>>> -
>>>>     By default, the back-end must complete all inflight I/O 
>>>> requests for the
>>>>     specified vring before stopping it.
>>>>       If the ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT`` 
>>>> protocol
>>>> -  feature has been negotiated, the back-end may suspend in-flight I/O
>>>> +  feature has been negotiated, using request payload's *num* field,
>>>> +  when *num* is set to 1, QEMU can tell the back-end to suspend 
>>>> in- flight I/O
>>>>     requests and record them as described in :ref:`Inflight I/O 
>>>> tracking
>>>>     <inflight_io_tracking>` instead of completing them before 
>>>> stopping the vring.
>>>>     How to suspend an in-flight request depends on the 
>>>> implementation of the back-end
>>>> diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
>>>> index a3ecd83f54..522e9d34b5 100644
>>>> --- a/hw/block/vhost-user-blk.c
>>>> +++ b/hw/block/vhost-user-blk.c
>>>> @@ -134,10 +134,7 @@ static bool 
>>>> vhost_user_blk_inflight_needed(void *opaque)
>>>>   {
>>>>       struct VHostUserBlk *s = opaque;
>>>>   -    bool inflight_migration = virtio_has_feature(s- 
>>>> >dev.protocol_features,
>>>> - VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
>>>> -
>>>> -    return inflight_migration;
>>>> +    return s->inflight_migration;
>>>>   }
>>>>     @@ -232,11 +229,14 @@ static int 
>>>> vhost_user_blk_stop(VirtIODevice *vdev)
>>>>           return 0;
>>>>       }
>>>>   +    bool skip_drain = vhost_user_blk_inflight_needed(s) &&
>>>
>>> nit: declarations on top
>>>
>>>> + runstate_check(RUN_STATE_FINISH_MIGRATE);
>>>> +
>>>>       force_stop = s->skip_get_vring_base_on_force_shutdown &&
>>>>                    qemu_force_shutdown_requested();
>>>>         ret = force_stop ? vhost_dev_force_stop(&s->dev, vdev, true) :
>>>> -                       vhost_dev_stop(&s->dev, vdev, true, false);
>>>> +                       vhost_dev_stop(&s->dev, vdev, true, 
>>>> skip_drain);
>>>>         if (k->set_guest_notifiers(qbus->parent, s->dev.nvqs, 
>>>> false) < 0) {
>>>>           error_report("vhost guest notifier cleanup failed: %d", 
>>>> ret);
>>>> @@ -364,7 +364,6 @@ static int vhost_user_blk_connect(DeviceState 
>>>> *dev, Error **errp)
>>>>       vhost_dev_set_config_notifier(&s->dev, &blk_ops);
>>>>         s->vhost_user.supports_config = true;
>>>> -    s->vhost_user.supports_inflight_migration = 
>>>> s->inflight_migration;
>>>>       ret = vhost_dev_init(&s->dev, &s->vhost_user, 
>>>> VHOST_BACKEND_TYPE_USER, 0,
>>>>                            errp);
>>>>       if (ret < 0) {
>>>> @@ -580,10 +579,29 @@ static struct vhost_dev 
>>>> *vhost_user_blk_get_vhost(VirtIODevice *vdev)
>>>>       return &s->dev;
>>>>   }
>>>>   +static bool vhost_user_blk_pre_save(void *opaque, Error **errp)
>>>> +{
>>>> +    VHostUserBlk *s = VHOST_USER_BLK(opaque);
>>>> +
>>>> +    bool inflight_migration_enabled = virtio_has_feature(
>>>> +        s->dev.protocol_features,
>>>> +        VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
>>>> +    if (vhost_user_blk_inflight_needed(s) && ! 
>>>> inflight_migration_enabled) {
>>>> +        error_setg(errp, "can't migrate vhost-user-blk device: "
>>>> +                         "backend doesn't support "
>>>> + "VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT "
>>>> +                         "protocol feature");
>>>> +        return false;
>>>> +    }
>>>> +
>>>> +    return true;
>>>> +}
>>>> +
>>>>   static const VMStateDescription vmstate_vhost_user_blk_inflight = {
>>>>       .name = "vhost-user-blk/inflight",
>>>>       .version_id = 1,
>>>>       .needed = vhost_user_blk_inflight_needed,
>>>> +    .pre_save_errp = vhost_user_blk_pre_save,
>>>>       .fields = (const VMStateField[]) {
>>>>           VMSTATE_VHOST_INFLIGHT_REGION(inflight, VHostUserBlk),
>>>>           VMSTATE_END_OF_LIST()
>>>> diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c
>>>> index bb8f8eab77..ed95ec7523 100644
>>>> --- a/hw/virtio/vhost-user.c
>>>> +++ b/hw/virtio/vhost-user.c
>>>> @@ -2225,8 +2225,7 @@ static int vhost_user_backend_init(struct 
>>>> vhost_dev *dev, void *opaque,
>>>>               }
>>>>           }
>>>>   -        if (!u->user->supports_inflight_migration ||
>>>> -            !virtio_has_feature(protocol_features,
>>>> +        if (!virtio_has_feature(protocol_features,
>>>> VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD)) {
>>>>               protocol_features &= ~(1ULL <<
>>>> VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
>>>> diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
>>>> index ac4d6fca73..54e4ce2660 100644
>>>> --- a/hw/virtio/vhost.c
>>>> +++ b/hw/virtio/vhost.c
>>>> @@ -1398,6 +1398,7 @@ static int do_vhost_virtqueue_stop(struct 
>>>> vhost_dev *dev,
>>>>       int vhost_vq_index = dev->vhost_ops->vhost_get_vq_index(dev, 
>>>> idx);
>>>>       struct vhost_vring_state state = {
>>>>           .index = vhost_vq_index,
>>>> +        .num = skip_drain,
>>>>       };
>>>>       int r = 0;
>>>>   diff --git a/include/hw/virtio/vhost-user.h b/include/hw/virtio/ 
>>>> vhost-user.h
>>>> index 53fe996686..c95bad5ddc 100644
>>>> --- a/include/hw/virtio/vhost-user.h
>>>> +++ b/include/hw/virtio/vhost-user.h
>>>> @@ -69,7 +69,6 @@ typedef struct VhostUserState {
>>>>       GPtrArray *notifiers;
>>>>       int memory_slots;
>>>>       bool supports_config;
>>>> -    bool supports_inflight_migration;
>>>>   } VhostUserState;
>>>>     /**
>>>
>

Re: [PATCH v3 5/5] vhost-user: add skip_drain param to GET_VRING_BASE

[Raphael Norwitz]

On 4/24/26 4:57 AM, Alexandr Moshkov wrote:
> Thanks for reply!
> 
> On 4/23/26 19:15, Raphael Norwitz wrote:
>>
>>
>> On 4/16/26 5:26 AM, Alexandr Moshkov wrote:
>>> Greetings! Thanks for reply!
>>>
>>>
>>> On 4/15/26 20:21, Raphael Norwitz wrote:
>>>> I’m not sure I like using the num field in vhost_vring_state to set 
>>>> magic values which affect protocol behavior for GET_VRING_BASE. It 
>>>> feels like a hack to me. I would think the proper solution if we 
>>>> want to support migration from new to old would be to have new use a 
>>>> different new message entirely. Can we do that?
>>>
>>> I think we can, but I thought at first that this will be almost a 
>>> complete copy of GET_VRING_BASE message, with the exception of 
>>> waiting for drain of requests, so I choose to expand existing message.
>>>
>>
>> I think a new message would be cleaner. Anyone else have thoughts?
>>
>>> If this is not an appropriate approach, is it better to make a new 
>>> message like GET_VRING_BASE or a separate message used together with 
>>> default GET_VRING_BASE message (for example, message for setting some 
>>> kind of status on the server)? What should I name this new message?
>>>
>>
>> Maybe GET_VRING_BASE_SKIP_DRAIN? How would a separate message used 
>> together with default GET_VRING_BASE message work? 
> 
> I was thinking about a message something like SET_VRING_ENABLE - for 
> example SET_SKIP_DRAIN_ENABLE, that would enable/disable some state 
> (skip_drain) in the backend.

I'd need to see the flow in more detail but sounds promising.

> 
>>
>>> I also fix other comments, thanks!
>>>
>>>>
>>>> On 3/30/26 11:52 AM, Alexandr Moshkov wrote:
>>>>> In case of migration of QEMU from the new version (where the
>>>>> inllight-migration parameter is present), to the old one (where it is
>>>>
>>>> nit: spelling inllight-migration
>>>>
>>>>> absent) there is no way to disable this feature on the backend during
>>>>> runtime.
>>>>>
>>>>> This commit slightly changes the semantics of the protocol feature
>>>>> VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT. Enabling this feature
>>>>> adds a new parameter for GET_VRING_BASE, which allows to control the
>>>>> drain in-flight requests on the backend.
>>>>> Thus, QEMU will be able to turn this feature on GET_VRING_BASE off and
>>>>> on anytime.
>>>>>
>>>>> In vhost-user-blk use inflight_migration param to enable skip_drain to
>>>>> suspend in-flight I/O requests, and then migrate them throught 
>>>>> inflight
>>>>> subsection.
>>>>>
>>>>> Also now QEMU will always try to setup
>>>>> VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT protocol featrue with
>>>>> backend. This will allow to use skip_drain parameter on GET_VRING_BASE
>>>>> message.
>>>>>
>>>>> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
>>>>> Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
>>>>> ---
>>>>>   docs/interop/vhost-user.rst    |  6 ++----
>>>>>   hw/block/vhost-user-blk.c      | 30 ++++++++++++++++++++++++------
>>>>>   hw/virtio/vhost-user.c         |  3 +--
>>>>>   hw/virtio/vhost.c              |  1 +
>>>>>   include/hw/virtio/vhost-user.h |  1 -
>>>>>   5 files changed, 28 insertions(+), 13 deletions(-)
>>>>>
>>>>> diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
>>>>> index bfa75ff9a3..63efc87264 100644
>>>>> --- a/docs/interop/vhost-user.rst
>>>>> +++ b/docs/interop/vhost-user.rst
>>>>> @@ -1255,14 +1255,12 @@ Front-end message types
>>>>>     *suspended*, see :ref:`Suspended device state
>>>>>     <suspended_device_state>`.
>>>>>   -  The request payload's *num* field is currently reserved and 
>>>>> must be
>>>>> -  set to 0.
>>>>> -
>>>>>     By default, the back-end must complete all inflight I/O 
>>>>> requests for the
>>>>>     specified vring before stopping it.
>>>>>       If the ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT`` 
>>>>> protocol
>>>>> -  feature has been negotiated, the back-end may suspend in-flight I/O
>>>>> +  feature has been negotiated, using request payload's *num* field,
>>>>> +  when *num* is set to 1, QEMU can tell the back-end to suspend 
>>>>> in- flight I/O
>>>>>     requests and record them as described in :ref:`Inflight I/O 
>>>>> tracking
>>>>>     <inflight_io_tracking>` instead of completing them before 
>>>>> stopping the vring.
>>>>>     How to suspend an in-flight request depends on the 
>>>>> implementation of the back-end
>>>>> diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
>>>>> index a3ecd83f54..522e9d34b5 100644
>>>>> --- a/hw/block/vhost-user-blk.c
>>>>> +++ b/hw/block/vhost-user-blk.c
>>>>> @@ -134,10 +134,7 @@ static bool 
>>>>> vhost_user_blk_inflight_needed(void *opaque)
>>>>>   {
>>>>>       struct VHostUserBlk *s = opaque;
>>>>>   -    bool inflight_migration = virtio_has_feature(s- 
>>>>> >dev.protocol_features,
>>>>> - VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
>>>>> -
>>>>> -    return inflight_migration;
>>>>> +    return s->inflight_migration;
>>>>>   }
>>>>>     @@ -232,11 +229,14 @@ static int 
>>>>> vhost_user_blk_stop(VirtIODevice *vdev)
>>>>>           return 0;
>>>>>       }
>>>>>   +    bool skip_drain = vhost_user_blk_inflight_needed(s) &&
>>>>
>>>> nit: declarations on top
>>>>
>>>>> + runstate_check(RUN_STATE_FINISH_MIGRATE);
>>>>> +
>>>>>       force_stop = s->skip_get_vring_base_on_force_shutdown &&
>>>>>                    qemu_force_shutdown_requested();
>>>>>         ret = force_stop ? vhost_dev_force_stop(&s->dev, vdev, true) :
>>>>> -                       vhost_dev_stop(&s->dev, vdev, true, false);
>>>>> +                       vhost_dev_stop(&s->dev, vdev, true, 
>>>>> skip_drain);
>>>>>         if (k->set_guest_notifiers(qbus->parent, s->dev.nvqs, 
>>>>> false) < 0) {
>>>>>           error_report("vhost guest notifier cleanup failed: %d", 
>>>>> ret);
>>>>> @@ -364,7 +364,6 @@ static int vhost_user_blk_connect(DeviceState 
>>>>> *dev, Error **errp)
>>>>>       vhost_dev_set_config_notifier(&s->dev, &blk_ops);
>>>>>         s->vhost_user.supports_config = true;
>>>>> -    s->vhost_user.supports_inflight_migration = s- 
>>>>> >inflight_migration;
>>>>>       ret = vhost_dev_init(&s->dev, &s->vhost_user, 
>>>>> VHOST_BACKEND_TYPE_USER, 0,
>>>>>                            errp);
>>>>>       if (ret < 0) {
>>>>> @@ -580,10 +579,29 @@ static struct vhost_dev 
>>>>> *vhost_user_blk_get_vhost(VirtIODevice *vdev)
>>>>>       return &s->dev;
>>>>>   }
>>>>>   +static bool vhost_user_blk_pre_save(void *opaque, Error **errp)
>>>>> +{
>>>>> +    VHostUserBlk *s = VHOST_USER_BLK(opaque);
>>>>> +
>>>>> +    bool inflight_migration_enabled = virtio_has_feature(
>>>>> +        s->dev.protocol_features,
>>>>> +        VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
>>>>> +    if (vhost_user_blk_inflight_needed(s) && ! 
>>>>> inflight_migration_enabled) {
>>>>> +        error_setg(errp, "can't migrate vhost-user-blk device: "
>>>>> +                         "backend doesn't support "
>>>>> + "VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT "
>>>>> +                         "protocol feature");
>>>>> +        return false;
>>>>> +    }
>>>>> +
>>>>> +    return true;
>>>>> +}
>>>>> +
>>>>>   static const VMStateDescription vmstate_vhost_user_blk_inflight = {
>>>>>       .name = "vhost-user-blk/inflight",
>>>>>       .version_id = 1,
>>>>>       .needed = vhost_user_blk_inflight_needed,
>>>>> +    .pre_save_errp = vhost_user_blk_pre_save,
>>>>>       .fields = (const VMStateField[]) {
>>>>>           VMSTATE_VHOST_INFLIGHT_REGION(inflight, VHostUserBlk),
>>>>>           VMSTATE_END_OF_LIST()
>>>>> diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c
>>>>> index bb8f8eab77..ed95ec7523 100644
>>>>> --- a/hw/virtio/vhost-user.c
>>>>> +++ b/hw/virtio/vhost-user.c
>>>>> @@ -2225,8 +2225,7 @@ static int vhost_user_backend_init(struct 
>>>>> vhost_dev *dev, void *opaque,
>>>>>               }
>>>>>           }
>>>>>   -        if (!u->user->supports_inflight_migration ||
>>>>> -            !virtio_has_feature(protocol_features,
>>>>> +        if (!virtio_has_feature(protocol_features,
>>>>> VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD)) {
>>>>>               protocol_features &= ~(1ULL <<
>>>>> VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
>>>>> diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
>>>>> index ac4d6fca73..54e4ce2660 100644
>>>>> --- a/hw/virtio/vhost.c
>>>>> +++ b/hw/virtio/vhost.c
>>>>> @@ -1398,6 +1398,7 @@ static int do_vhost_virtqueue_stop(struct 
>>>>> vhost_dev *dev,
>>>>>       int vhost_vq_index = dev->vhost_ops->vhost_get_vq_index(dev, 
>>>>> idx);
>>>>>       struct vhost_vring_state state = {
>>>>>           .index = vhost_vq_index,
>>>>> +        .num = skip_drain,
>>>>>       };
>>>>>       int r = 0;
>>>>>   diff --git a/include/hw/virtio/vhost-user.h b/include/hw/virtio/ 
>>>>> vhost-user.h
>>>>> index 53fe996686..c95bad5ddc 100644
>>>>> --- a/include/hw/virtio/vhost-user.h
>>>>> +++ b/include/hw/virtio/vhost-user.h
>>>>> @@ -69,7 +69,6 @@ typedef struct VhostUserState {
>>>>>       GPtrArray *notifiers;
>>>>>       int memory_slots;
>>>>>       bool supports_config;
>>>>> -    bool supports_inflight_migration;
>>>>>   } VhostUserState;
>>>>>     /**
>>>>
>>

Re: [PATCH v3 5/5] vhost-user: add skip_drain param to GET_VRING_BASE

[Alexandr Moshkov]

On 4/24/26 17:52, Raphael Norwitz wrote:
>
>
> On 4/24/26 4:57 AM, Alexandr Moshkov wrote:
>> Thanks for reply!
>>
>> On 4/23/26 19:15, Raphael Norwitz wrote:
>>>
>>>
>>> On 4/16/26 5:26 AM, Alexandr Moshkov wrote:
>>>> Greetings! Thanks for reply!
>>>>
>>>>
>>>> On 4/15/26 20:21, Raphael Norwitz wrote:
>>>>> I’m not sure I like using the num field in vhost_vring_state to 
>>>>> set magic values which affect protocol behavior for 
>>>>> GET_VRING_BASE. It feels like a hack to me. I would think the 
>>>>> proper solution if we want to support migration from new to old 
>>>>> would be to have new use a different new message entirely. Can we 
>>>>> do that?
>>>>
>>>> I think we can, but I thought at first that this will be almost a 
>>>> complete copy of GET_VRING_BASE message, with the exception of 
>>>> waiting for drain of requests, so I choose to expand existing message.
>>>>
>>>
>>> I think a new message would be cleaner. Anyone else have thoughts?
>>>
>>>> If this is not an appropriate approach, is it better to make a new 
>>>> message like GET_VRING_BASE or a separate message used together 
>>>> with default GET_VRING_BASE message (for example, message for 
>>>> setting some kind of status on the server)? What should I name this 
>>>> new message?
>>>>
>>>
>>> Maybe GET_VRING_BASE_SKIP_DRAIN? How would a separate message used 
>>> together with default GET_VRING_BASE message work? 
>>
>> I was thinking about a message something like SET_VRING_ENABLE - for 
>> example SET_SKIP_DRAIN_ENABLE, that would enable/disable some state 
>> (skip_drain) in the backend.
>
> I'd need to see the flow in more detail but sounds promising.

On migration vhost-user-blk firstly send SET_SKIP_DRAIN_ENABLE with num 
= 1 message if `inflight-migration` device parameter 
and VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT enabled. Then send 
GET_VRING_BASE and continue work as usual.

On SET_SKIP_DRAIN_ENABLE backend sets inner state (skip_drain) so when 
GET_VRING_BASE is called and skip_drain is true the drain will skipped.

What do you think?

>
>>
>>>
>>>> I also fix other comments, thanks!
>>>>
>>>>>
>>>>> On 3/30/26 11:52 AM, Alexandr Moshkov wrote:
>>>>>> In case of migration of QEMU from the new version (where the
>>>>>> inllight-migration parameter is present), to the old one (where 
>>>>>> it is
>>>>>
>>>>> nit: spelling inllight-migration
>>>>>
>>>>>> absent) there is no way to disable this feature on the backend 
>>>>>> during
>>>>>> runtime.
>>>>>>
>>>>>> This commit slightly changes the semantics of the protocol feature
>>>>>> VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT. Enabling this feature
>>>>>> adds a new parameter for GET_VRING_BASE, which allows to control the
>>>>>> drain in-flight requests on the backend.
>>>>>> Thus, QEMU will be able to turn this feature on GET_VRING_BASE 
>>>>>> off and
>>>>>> on anytime.
>>>>>>
>>>>>> In vhost-user-blk use inflight_migration param to enable 
>>>>>> skip_drain to
>>>>>> suspend in-flight I/O requests, and then migrate them throught 
>>>>>> inflight
>>>>>> subsection.
>>>>>>
>>>>>> Also now QEMU will always try to setup
>>>>>> VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT protocol featrue with
>>>>>> backend. This will allow to use skip_drain parameter on 
>>>>>> GET_VRING_BASE
>>>>>> message.
>>>>>>
>>>>>> Reviewed-by: Vladimir Sementsov-Ogievskiy 
>>>>>> <vsementsov@yandex-team.ru>
>>>>>> Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
>>>>>> ---
>>>>>>   docs/interop/vhost-user.rst    |  6 ++----
>>>>>>   hw/block/vhost-user-blk.c      | 30 ++++++++++++++++++++++++------
>>>>>>   hw/virtio/vhost-user.c         |  3 +--
>>>>>>   hw/virtio/vhost.c              |  1 +
>>>>>>   include/hw/virtio/vhost-user.h |  1 -
>>>>>>   5 files changed, 28 insertions(+), 13 deletions(-)
>>>>>>
>>>>>> diff --git a/docs/interop/vhost-user.rst 
>>>>>> b/docs/interop/vhost-user.rst
>>>>>> index bfa75ff9a3..63efc87264 100644
>>>>>> --- a/docs/interop/vhost-user.rst
>>>>>> +++ b/docs/interop/vhost-user.rst
>>>>>> @@ -1255,14 +1255,12 @@ Front-end message types
>>>>>>     *suspended*, see :ref:`Suspended device state
>>>>>>     <suspended_device_state>`.
>>>>>>   -  The request payload's *num* field is currently reserved and 
>>>>>> must be
>>>>>> -  set to 0.
>>>>>> -
>>>>>>     By default, the back-end must complete all inflight I/O 
>>>>>> requests for the
>>>>>>     specified vring before stopping it.
>>>>>>       If the ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT`` 
>>>>>> protocol
>>>>>> -  feature has been negotiated, the back-end may suspend 
>>>>>> in-flight I/O
>>>>>> +  feature has been negotiated, using request payload's *num* field,
>>>>>> +  when *num* is set to 1, QEMU can tell the back-end to suspend 
>>>>>> in- flight I/O
>>>>>>     requests and record them as described in :ref:`Inflight I/O 
>>>>>> tracking
>>>>>>     <inflight_io_tracking>` instead of completing them before 
>>>>>> stopping the vring.
>>>>>>     How to suspend an in-flight request depends on the 
>>>>>> implementation of the back-end
>>>>>> diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
>>>>>> index a3ecd83f54..522e9d34b5 100644
>>>>>> --- a/hw/block/vhost-user-blk.c
>>>>>> +++ b/hw/block/vhost-user-blk.c
>>>>>> @@ -134,10 +134,7 @@ static bool 
>>>>>> vhost_user_blk_inflight_needed(void *opaque)
>>>>>>   {
>>>>>>       struct VHostUserBlk *s = opaque;
>>>>>>   -    bool inflight_migration = virtio_has_feature(s- 
>>>>>> >dev.protocol_features,
>>>>>> - VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
>>>>>> -
>>>>>> -    return inflight_migration;
>>>>>> +    return s->inflight_migration;
>>>>>>   }
>>>>>>     @@ -232,11 +229,14 @@ static int 
>>>>>> vhost_user_blk_stop(VirtIODevice *vdev)
>>>>>>           return 0;
>>>>>>       }
>>>>>>   +    bool skip_drain = vhost_user_blk_inflight_needed(s) &&
>>>>>
>>>>> nit: declarations on top
>>>>>
>>>>>> + runstate_check(RUN_STATE_FINISH_MIGRATE);
>>>>>> +
>>>>>>       force_stop = s->skip_get_vring_base_on_force_shutdown &&
>>>>>>                    qemu_force_shutdown_requested();
>>>>>>         ret = force_stop ? vhost_dev_force_stop(&s->dev, vdev, 
>>>>>> true) :
>>>>>> -                       vhost_dev_stop(&s->dev, vdev, true, false);
>>>>>> +                       vhost_dev_stop(&s->dev, vdev, true, 
>>>>>> skip_drain);
>>>>>>         if (k->set_guest_notifiers(qbus->parent, s->dev.nvqs, 
>>>>>> false) < 0) {
>>>>>>           error_report("vhost guest notifier cleanup failed: %d", 
>>>>>> ret);
>>>>>> @@ -364,7 +364,6 @@ static int vhost_user_blk_connect(DeviceState 
>>>>>> *dev, Error **errp)
>>>>>>       vhost_dev_set_config_notifier(&s->dev, &blk_ops);
>>>>>>         s->vhost_user.supports_config = true;
>>>>>> -    s->vhost_user.supports_inflight_migration = s- 
>>>>>> >inflight_migration;
>>>>>>       ret = vhost_dev_init(&s->dev, &s->vhost_user, 
>>>>>> VHOST_BACKEND_TYPE_USER, 0,
>>>>>>                            errp);
>>>>>>       if (ret < 0) {
>>>>>> @@ -580,10 +579,29 @@ static struct vhost_dev 
>>>>>> *vhost_user_blk_get_vhost(VirtIODevice *vdev)
>>>>>>       return &s->dev;
>>>>>>   }
>>>>>>   +static bool vhost_user_blk_pre_save(void *opaque, Error **errp)
>>>>>> +{
>>>>>> +    VHostUserBlk *s = VHOST_USER_BLK(opaque);
>>>>>> +
>>>>>> +    bool inflight_migration_enabled = virtio_has_feature(
>>>>>> +        s->dev.protocol_features,
>>>>>> +        VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
>>>>>> +    if (vhost_user_blk_inflight_needed(s) && ! 
>>>>>> inflight_migration_enabled) {
>>>>>> +        error_setg(errp, "can't migrate vhost-user-blk device: "
>>>>>> +                         "backend doesn't support "
>>>>>> + "VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT "
>>>>>> +                         "protocol feature");
>>>>>> +        return false;
>>>>>> +    }
>>>>>> +
>>>>>> +    return true;
>>>>>> +}
>>>>>> +
>>>>>>   static const VMStateDescription vmstate_vhost_user_blk_inflight 
>>>>>> = {
>>>>>>       .name = "vhost-user-blk/inflight",
>>>>>>       .version_id = 1,
>>>>>>       .needed = vhost_user_blk_inflight_needed,
>>>>>> +    .pre_save_errp = vhost_user_blk_pre_save,
>>>>>>       .fields = (const VMStateField[]) {
>>>>>>           VMSTATE_VHOST_INFLIGHT_REGION(inflight, VHostUserBlk),
>>>>>>           VMSTATE_END_OF_LIST()
>>>>>> diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c
>>>>>> index bb8f8eab77..ed95ec7523 100644
>>>>>> --- a/hw/virtio/vhost-user.c
>>>>>> +++ b/hw/virtio/vhost-user.c
>>>>>> @@ -2225,8 +2225,7 @@ static int vhost_user_backend_init(struct 
>>>>>> vhost_dev *dev, void *opaque,
>>>>>>               }
>>>>>>           }
>>>>>>   -        if (!u->user->supports_inflight_migration ||
>>>>>> -            !virtio_has_feature(protocol_features,
>>>>>> +        if (!virtio_has_feature(protocol_features,
>>>>>> VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD)) {
>>>>>>               protocol_features &= ~(1ULL <<
>>>>>> VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
>>>>>> diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
>>>>>> index ac4d6fca73..54e4ce2660 100644
>>>>>> --- a/hw/virtio/vhost.c
>>>>>> +++ b/hw/virtio/vhost.c
>>>>>> @@ -1398,6 +1398,7 @@ static int do_vhost_virtqueue_stop(struct 
>>>>>> vhost_dev *dev,
>>>>>>       int vhost_vq_index = 
>>>>>> dev->vhost_ops->vhost_get_vq_index(dev, idx);
>>>>>>       struct vhost_vring_state state = {
>>>>>>           .index = vhost_vq_index,
>>>>>> +        .num = skip_drain,
>>>>>>       };
>>>>>>       int r = 0;
>>>>>>   diff --git a/include/hw/virtio/vhost-user.h 
>>>>>> b/include/hw/virtio/ vhost-user.h
>>>>>> index 53fe996686..c95bad5ddc 100644
>>>>>> --- a/include/hw/virtio/vhost-user.h
>>>>>> +++ b/include/hw/virtio/vhost-user.h
>>>>>> @@ -69,7 +69,6 @@ typedef struct VhostUserState {
>>>>>>       GPtrArray *notifiers;
>>>>>>       int memory_slots;
>>>>>>       bool supports_config;
>>>>>> -    bool supports_inflight_migration;
>>>>>>   } VhostUserState;
>>>>>>     /**
>>>>>
>>>
>

Re: [PATCH v3 0/5] vhost-user-blk: fix compatibility with older qemu versions

[Alexandr Moshkov]

Ping :)

On 4/20/26 16:11, Alexandr Moshkov wrote:
> Hello! Another gentle ping :)
>
> On 4/6/26 14:51, Alexandr Moshkov wrote:
>> Ping :)
>>
>> On 3/30/26 14:52, Alexandr Moshkov wrote:
>>> v2 -> v3:
>>> - fix complile problems
>>> - add assert check in do_vhost_virtio_stop
>>> - make inflight-migration property mutable
>>>
>>> v1 -> v2:
>>> - reorganize commits: make refactor commits first, then core 
>>> semantic change
>>> - add additional pre_save check for inflight migration possibility
>>>
>>> ---
>>>
>>> This is a small continuation of my series about inflight migration 
>>> for vhost-user-blk.
>>>
>>> This series is designed to solve the problem of compatibility with 
>>> older versions of qemu, where this feature has not yet been 
>>> introduced (for example, if we want to downgrade versions due to 
>>> some problems).
>>>
>>> In the current version for vhost-user-blk, this feature is enabled 
>>> using the parameter and further migration of the inflight region 
>>> will depend on whether the vhost-user has accepted the new protocol 
>>> feature or not. This creates an inconvenient dependency, because 
>>> there is no way to disable this feature without reconnecting to the 
>>> backend.
>>> This series slightly changes the semantics of the introduced 
>>> protocol feature VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT. 
>>> Enabling this feature adds a new parameter for GET_VRING_BASE 
>>> message - skip_drain, which allows to control drain in-flight 
>>> requests on the backend.
>>>
>>> Thus, user can enable or disable inflight-migration param for 
>>> vhost-user-blk to maintain compatibility with older versions of QEMU.
>>>
>>> Alexandr Moshkov (5):
>>>    vhost-user.rst: fix typo
>>>    vhost-user-blk: make inflight-migration prop mutable
>>>    vhost-user: add skip_drain param to do_vhost_virtqueue_stop
>>>    vhost-user-blk: move inflight_needed higher
>>>    vhost-user: add skip_drain param to GET_VRING_BASE
>>>
>>>   backends/cryptodev-vhost.c     |  2 +-
>>>   backends/vhost-user.c          |  2 +-
>>>   docs/interop/vhost-user.rst    |  8 +++----
>>>   hw/block/vhost-user-blk.c      | 43 
>>> +++++++++++++++++++++++++++-------
>>>   hw/net/vhost_net.c             |  9 +++----
>>>   hw/scsi/vhost-scsi-common.c    |  2 +-
>>>   hw/virtio/vdpa-dev.c           |  2 +-
>>>   hw/virtio/vhost-user-base.c    |  2 +-
>>>   hw/virtio/vhost-user-fs.c      |  2 +-
>>>   hw/virtio/vhost-user-scmi.c    |  2 +-
>>>   hw/virtio/vhost-user.c         |  3 +--
>>>   hw/virtio/vhost-vsock-common.c |  2 +-
>>>   hw/virtio/vhost.c              | 29 ++++++++++++++++-------
>>>   include/hw/virtio/vhost-user.h |  1 -
>>>   include/hw/virtio/vhost.h      |  7 ++++--
>>>   15 files changed, 76 insertions(+), 40 deletions(-)
>>>

Re: [PATCH v3 5/5] vhost-user: add skip_drain param to GET_VRING_BASE

[Alexandr Moshkov]

Gentle ping :)

On 4/27/26 16:15, Alexandr Moshkov wrote:
>
> On 4/24/26 17:52, Raphael Norwitz wrote:
>>
>>
>> On 4/24/26 4:57 AM, Alexandr Moshkov wrote:
>>> Thanks for reply!
>>>
>>> On 4/23/26 19:15, Raphael Norwitz wrote:
>>>>
>>>>
>>>> On 4/16/26 5:26 AM, Alexandr Moshkov wrote:
>>>>> Greetings! Thanks for reply!
>>>>>
>>>>>
>>>>> On 4/15/26 20:21, Raphael Norwitz wrote:
>>>>>> I’m not sure I like using the num field in vhost_vring_state to 
>>>>>> set magic values which affect protocol behavior for 
>>>>>> GET_VRING_BASE. It feels like a hack to me. I would think the 
>>>>>> proper solution if we want to support migration from new to old 
>>>>>> would be to have new use a different new message entirely. Can we 
>>>>>> do that?
>>>>>
>>>>> I think we can, but I thought at first that this will be almost a 
>>>>> complete copy of GET_VRING_BASE message, with the exception of 
>>>>> waiting for drain of requests, so I choose to expand existing 
>>>>> message.
>>>>>
>>>>
>>>> I think a new message would be cleaner. Anyone else have thoughts?
>>>>
>>>>> If this is not an appropriate approach, is it better to make a new 
>>>>> message like GET_VRING_BASE or a separate message used together 
>>>>> with default GET_VRING_BASE message (for example, message for 
>>>>> setting some kind of status on the server)? What should I name 
>>>>> this new message?
>>>>>
>>>>
>>>> Maybe GET_VRING_BASE_SKIP_DRAIN? How would a separate message used 
>>>> together with default GET_VRING_BASE message work? 
>>>
>>> I was thinking about a message something like SET_VRING_ENABLE - for 
>>> example SET_SKIP_DRAIN_ENABLE, that would enable/disable some state 
>>> (skip_drain) in the backend.
>>
>> I'd need to see the flow in more detail but sounds promising.
>
> On migration vhost-user-blk firstly send SET_SKIP_DRAIN_ENABLE with 
> num = 1 message if `inflight-migration` device parameter 
> and VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT enabled. Then send 
> GET_VRING_BASE and continue work as usual.
>
> On SET_SKIP_DRAIN_ENABLE backend sets inner state (skip_drain) so when 
> GET_VRING_BASE is called and skip_drain is true the drain will skipped.
>
> What do you think?
>
>>
>>>
>>>>
>>>>> I also fix other comments, thanks!
>>>>>
>>>>>>
>>>>>> On 3/30/26 11:52 AM, Alexandr Moshkov wrote:
>>>>>>> In case of migration of QEMU from the new version (where the
>>>>>>> inllight-migration parameter is present), to the old one (where 
>>>>>>> it is
>>>>>>
>>>>>> nit: spelling inllight-migration
>>>>>>
>>>>>>> absent) there is no way to disable this feature on the backend 
>>>>>>> during
>>>>>>> runtime.
>>>>>>>
>>>>>>> This commit slightly changes the semantics of the protocol feature
>>>>>>> VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT. Enabling this 
>>>>>>> feature
>>>>>>> adds a new parameter for GET_VRING_BASE, which allows to control 
>>>>>>> the
>>>>>>> drain in-flight requests on the backend.
>>>>>>> Thus, QEMU will be able to turn this feature on GET_VRING_BASE 
>>>>>>> off and
>>>>>>> on anytime.
>>>>>>>
>>>>>>> In vhost-user-blk use inflight_migration param to enable 
>>>>>>> skip_drain to
>>>>>>> suspend in-flight I/O requests, and then migrate them throught 
>>>>>>> inflight
>>>>>>> subsection.
>>>>>>>
>>>>>>> Also now QEMU will always try to setup
>>>>>>> VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT protocol featrue with
>>>>>>> backend. This will allow to use skip_drain parameter on 
>>>>>>> GET_VRING_BASE
>>>>>>> message.
>>>>>>>
>>>>>>> Reviewed-by: Vladimir Sementsov-Ogievskiy 
>>>>>>> <vsementsov@yandex-team.ru>
>>>>>>> Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
>>>>>>> ---
>>>>>>>   docs/interop/vhost-user.rst    |  6 ++----
>>>>>>>   hw/block/vhost-user-blk.c      | 30 
>>>>>>> ++++++++++++++++++++++++------
>>>>>>>   hw/virtio/vhost-user.c         |  3 +--
>>>>>>>   hw/virtio/vhost.c              |  1 +
>>>>>>>   include/hw/virtio/vhost-user.h |  1 -
>>>>>>>   5 files changed, 28 insertions(+), 13 deletions(-)
>>>>>>>
>>>>>>> diff --git a/docs/interop/vhost-user.rst 
>>>>>>> b/docs/interop/vhost-user.rst
>>>>>>> index bfa75ff9a3..63efc87264 100644
>>>>>>> --- a/docs/interop/vhost-user.rst
>>>>>>> +++ b/docs/interop/vhost-user.rst
>>>>>>> @@ -1255,14 +1255,12 @@ Front-end message types
>>>>>>>     *suspended*, see :ref:`Suspended device state
>>>>>>>     <suspended_device_state>`.
>>>>>>>   -  The request payload's *num* field is currently reserved and 
>>>>>>> must be
>>>>>>> -  set to 0.
>>>>>>> -
>>>>>>>     By default, the back-end must complete all inflight I/O 
>>>>>>> requests for the
>>>>>>>     specified vring before stopping it.
>>>>>>>       If the ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT`` 
>>>>>>> protocol
>>>>>>> -  feature has been negotiated, the back-end may suspend 
>>>>>>> in-flight I/O
>>>>>>> +  feature has been negotiated, using request payload's *num* 
>>>>>>> field,
>>>>>>> +  when *num* is set to 1, QEMU can tell the back-end to suspend 
>>>>>>> in- flight I/O
>>>>>>>     requests and record them as described in :ref:`Inflight I/O 
>>>>>>> tracking
>>>>>>>     <inflight_io_tracking>` instead of completing them before 
>>>>>>> stopping the vring.
>>>>>>>     How to suspend an in-flight request depends on the 
>>>>>>> implementation of the back-end
>>>>>>> diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
>>>>>>> index a3ecd83f54..522e9d34b5 100644
>>>>>>> --- a/hw/block/vhost-user-blk.c
>>>>>>> +++ b/hw/block/vhost-user-blk.c
>>>>>>> @@ -134,10 +134,7 @@ static bool 
>>>>>>> vhost_user_blk_inflight_needed(void *opaque)
>>>>>>>   {
>>>>>>>       struct VHostUserBlk *s = opaque;
>>>>>>>   -    bool inflight_migration = virtio_has_feature(s- 
>>>>>>> >dev.protocol_features,
>>>>>>> - VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
>>>>>>> -
>>>>>>> -    return inflight_migration;
>>>>>>> +    return s->inflight_migration;
>>>>>>>   }
>>>>>>>     @@ -232,11 +229,14 @@ static int 
>>>>>>> vhost_user_blk_stop(VirtIODevice *vdev)
>>>>>>>           return 0;
>>>>>>>       }
>>>>>>>   +    bool skip_drain = vhost_user_blk_inflight_needed(s) &&
>>>>>>
>>>>>> nit: declarations on top
>>>>>>
>>>>>>> + runstate_check(RUN_STATE_FINISH_MIGRATE);
>>>>>>> +
>>>>>>>       force_stop = s->skip_get_vring_base_on_force_shutdown &&
>>>>>>>                    qemu_force_shutdown_requested();
>>>>>>>         ret = force_stop ? vhost_dev_force_stop(&s->dev, vdev, 
>>>>>>> true) :
>>>>>>> -                       vhost_dev_stop(&s->dev, vdev, true, false);
>>>>>>> +                       vhost_dev_stop(&s->dev, vdev, true, 
>>>>>>> skip_drain);
>>>>>>>         if (k->set_guest_notifiers(qbus->parent, s->dev.nvqs, 
>>>>>>> false) < 0) {
>>>>>>>           error_report("vhost guest notifier cleanup failed: 
>>>>>>> %d", ret);
>>>>>>> @@ -364,7 +364,6 @@ static int 
>>>>>>> vhost_user_blk_connect(DeviceState *dev, Error **errp)
>>>>>>>       vhost_dev_set_config_notifier(&s->dev, &blk_ops);
>>>>>>>         s->vhost_user.supports_config = true;
>>>>>>> -    s->vhost_user.supports_inflight_migration = s- 
>>>>>>> >inflight_migration;
>>>>>>>       ret = vhost_dev_init(&s->dev, &s->vhost_user, 
>>>>>>> VHOST_BACKEND_TYPE_USER, 0,
>>>>>>>                            errp);
>>>>>>>       if (ret < 0) {
>>>>>>> @@ -580,10 +579,29 @@ static struct vhost_dev 
>>>>>>> *vhost_user_blk_get_vhost(VirtIODevice *vdev)
>>>>>>>       return &s->dev;
>>>>>>>   }
>>>>>>>   +static bool vhost_user_blk_pre_save(void *opaque, Error **errp)
>>>>>>> +{
>>>>>>> +    VHostUserBlk *s = VHOST_USER_BLK(opaque);
>>>>>>> +
>>>>>>> +    bool inflight_migration_enabled = virtio_has_feature(
>>>>>>> +        s->dev.protocol_features,
>>>>>>> + VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
>>>>>>> +    if (vhost_user_blk_inflight_needed(s) && ! 
>>>>>>> inflight_migration_enabled) {
>>>>>>> +        error_setg(errp, "can't migrate vhost-user-blk device: "
>>>>>>> +                         "backend doesn't support "
>>>>>>> + "VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT "
>>>>>>> +                         "protocol feature");
>>>>>>> +        return false;
>>>>>>> +    }
>>>>>>> +
>>>>>>> +    return true;
>>>>>>> +}
>>>>>>> +
>>>>>>>   static const VMStateDescription 
>>>>>>> vmstate_vhost_user_blk_inflight = {
>>>>>>>       .name = "vhost-user-blk/inflight",
>>>>>>>       .version_id = 1,
>>>>>>>       .needed = vhost_user_blk_inflight_needed,
>>>>>>> +    .pre_save_errp = vhost_user_blk_pre_save,
>>>>>>>       .fields = (const VMStateField[]) {
>>>>>>>           VMSTATE_VHOST_INFLIGHT_REGION(inflight, VHostUserBlk),
>>>>>>>           VMSTATE_END_OF_LIST()
>>>>>>> diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c
>>>>>>> index bb8f8eab77..ed95ec7523 100644
>>>>>>> --- a/hw/virtio/vhost-user.c
>>>>>>> +++ b/hw/virtio/vhost-user.c
>>>>>>> @@ -2225,8 +2225,7 @@ static int vhost_user_backend_init(struct 
>>>>>>> vhost_dev *dev, void *opaque,
>>>>>>>               }
>>>>>>>           }
>>>>>>>   -        if (!u->user->supports_inflight_migration ||
>>>>>>> -            !virtio_has_feature(protocol_features,
>>>>>>> +        if (!virtio_has_feature(protocol_features,
>>>>>>> VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD)) {
>>>>>>>               protocol_features &= ~(1ULL <<
>>>>>>> VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
>>>>>>> diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
>>>>>>> index ac4d6fca73..54e4ce2660 100644
>>>>>>> --- a/hw/virtio/vhost.c
>>>>>>> +++ b/hw/virtio/vhost.c
>>>>>>> @@ -1398,6 +1398,7 @@ static int do_vhost_virtqueue_stop(struct 
>>>>>>> vhost_dev *dev,
>>>>>>>       int vhost_vq_index = 
>>>>>>> dev->vhost_ops->vhost_get_vq_index(dev, idx);
>>>>>>>       struct vhost_vring_state state = {
>>>>>>>           .index = vhost_vq_index,
>>>>>>> +        .num = skip_drain,
>>>>>>>       };
>>>>>>>       int r = 0;
>>>>>>>   diff --git a/include/hw/virtio/vhost-user.h 
>>>>>>> b/include/hw/virtio/ vhost-user.h
>>>>>>> index 53fe996686..c95bad5ddc 100644
>>>>>>> --- a/include/hw/virtio/vhost-user.h
>>>>>>> +++ b/include/hw/virtio/vhost-user.h
>>>>>>> @@ -69,7 +69,6 @@ typedef struct VhostUserState {
>>>>>>>       GPtrArray *notifiers;
>>>>>>>       int memory_slots;
>>>>>>>       bool supports_config;
>>>>>>> -    bool supports_inflight_migration;
>>>>>>>   } VhostUserState;
>>>>>>>     /**
>>>>>>
>>>>
>>

Re: [PATCH v3 5/5] vhost-user: add skip_drain param to GET_VRING_BASE

[Raphael Norwitz]

Apologies for the late reply.

On 5/12/26 1:55 AM, Alexandr Moshkov wrote:
> Gentle ping :)
> 

[...]

>>>>> On 4/16/26 5:26 AM, Alexandr Moshkov wrote:
>>>>>> Greetings! Thanks for reply!
>>>>>>
>>>>>>
>>>>>> On 4/15/26 20:21, Raphael Norwitz wrote:
>>>>>>> I’m not sure I like using the num field in vhost_vring_state to 
>>>>>>> set magic values which affect protocol behavior for 
>>>>>>> GET_VRING_BASE. It feels like a hack to me. I would think the 
>>>>>>> proper solution if we want to support migration from new to old 
>>>>>>> would be to have new use a different new message entirely. Can we 
>>>>>>> do that?
>>>>>>
>>>>>> I think we can, but I thought at first that this will be almost a 
>>>>>> complete copy of GET_VRING_BASE message, with the exception of 
>>>>>> waiting for drain of requests, so I choose to expand existing 
>>>>>> message.
>>>>>>
>>>>>
>>>>> I think a new message would be cleaner. Anyone else have thoughts?
>>>>>
>>>>>> If this is not an appropriate approach, is it better to make a new 
>>>>>> message like GET_VRING_BASE or a separate message used together 
>>>>>> with default GET_VRING_BASE message (for example, message for 
>>>>>> setting some kind of status on the server)? What should I name 
>>>>>> this new message?
>>>>>>
>>>>>
>>>>> Maybe GET_VRING_BASE_SKIP_DRAIN? How would a separate message used 
>>>>> together with default GET_VRING_BASE message work? 
>>>>
>>>> I was thinking about a message something like SET_VRING_ENABLE - for 
>>>> example SET_SKIP_DRAIN_ENABLE, that would enable/disable some state 
>>>> (skip_drain) in the backend.
>>>
>>> I'd need to see the flow in more detail but sounds promising.
>>
>> On migration vhost-user-blk firstly send SET_SKIP_DRAIN_ENABLE with 
>> num = 1 message if `inflight-migration` device parameter 
>> and VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT enabled. Then send 
>> GET_VRING_BASE and continue work as usual.
>>
>> On SET_SKIP_DRAIN_ENABLE backend sets inner state (skip_drain) so when 
>> GET_VRING_BASE is called and skip_drain is true the drain will skipped.
>>
>> What do you think?
>>

I'm happy with that in theory but would like more clarity on the 
details. In particular, I'm not sure what you mean by "vhost-user-blk 
firstly send SET_SKIP_DRAIN_ENABLE". To add a new message I would think 
we would have to do a vhost-user protocol feature negotiation to gate it.

Also what are the precise semantics for SET_SKIP_DRAIN_ENABLE? Would it 
be sent on backend connect or when we're about to migrate?

I was hoping others would comment but at this point I'd suggest drafting 
the code and then we can re-review.

>>>
>>>>
>>>>>
>>>>>> I also fix other comments, thanks!
>>>>>>
>>>>>>>
>>>>>>> On 3/30/26 11:52 AM, Alexandr Moshkov wrote:
>>>>>>>> In case of migration of QEMU from the new version (where the
>>>>>>>> inllight-migration parameter is present), to the old one (where 
>>>>>>>> it is
>>>>>>>
>>>>>>> nit: spelling inllight-migration
>>>>>>>
>>>>>>>> absent) there is no way to disable this feature on the backend 
>>>>>>>> during
>>>>>>>> runtime.
>>>>>>>>
>>>>>>>> This commit slightly changes the semantics of the protocol feature
>>>>>>>> VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT. Enabling this 
>>>>>>>> feature
>>>>>>>> adds a new parameter for GET_VRING_BASE, which allows to control 
>>>>>>>> the
>>>>>>>> drain in-flight requests on the backend.
>>>>>>>> Thus, QEMU will be able to turn this feature on GET_VRING_BASE 
>>>>>>>> off and
>>>>>>>> on anytime.
>>>>>>>>
>>>>>>>> In vhost-user-blk use inflight_migration param to enable 
>>>>>>>> skip_drain to
>>>>>>>> suspend in-flight I/O requests, and then migrate them throught 
>>>>>>>> inflight
>>>>>>>> subsection.
>>>>>>>>
>>>>>>>> Also now QEMU will always try to setup
>>>>>>>> VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT protocol featrue with
>>>>>>>> backend. This will allow to use skip_drain parameter on 
>>>>>>>> GET_VRING_BASE
>>>>>>>> message.
>>>>>>>>
>>>>>>>> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex- 
>>>>>>>> team.ru>
>>>>>>>> Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
>>>>>>>> ---
>>>>>>>>   docs/interop/vhost-user.rst    |  6 ++----
>>>>>>>>   hw/block/vhost-user-blk.c      | 30 +++++++++++++++++++++++ 
>>>>>>>> +------
>>>>>>>>   hw/virtio/vhost-user.c         |  3 +--
>>>>>>>>   hw/virtio/vhost.c              |  1 +
>>>>>>>>   include/hw/virtio/vhost-user.h |  1 -
>>>>>>>>   5 files changed, 28 insertions(+), 13 deletions(-)
>>>>>>>>
>>>>>>>> diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost- 
>>>>>>>> user.rst
>>>>>>>> index bfa75ff9a3..63efc87264 100644
>>>>>>>> --- a/docs/interop/vhost-user.rst
>>>>>>>> +++ b/docs/interop/vhost-user.rst
>>>>>>>> @@ -1255,14 +1255,12 @@ Front-end message types
>>>>>>>>     *suspended*, see :ref:`Suspended device state
>>>>>>>>     <suspended_device_state>`.
>>>>>>>>   -  The request payload's *num* field is currently reserved and 
>>>>>>>> must be
>>>>>>>> -  set to 0.
>>>>>>>> -
>>>>>>>>     By default, the back-end must complete all inflight I/O 
>>>>>>>> requests for the
>>>>>>>>     specified vring before stopping it.
>>>>>>>>       If the ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT`` 
>>>>>>>> protocol
>>>>>>>> -  feature has been negotiated, the back-end may suspend in- 
>>>>>>>> flight I/O
>>>>>>>> +  feature has been negotiated, using request payload's *num* 
>>>>>>>> field,
>>>>>>>> +  when *num* is set to 1, QEMU can tell the back-end to suspend 
>>>>>>>> in- flight I/O
>>>>>>>>     requests and record them as described in :ref:`Inflight I/O 
>>>>>>>> tracking
>>>>>>>>     <inflight_io_tracking>` instead of completing them before 
>>>>>>>> stopping the vring.
>>>>>>>>     How to suspend an in-flight request depends on the 
>>>>>>>> implementation of the back-end
>>>>>>>> diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
>>>>>>>> index a3ecd83f54..522e9d34b5 100644
>>>>>>>> --- a/hw/block/vhost-user-blk.c
>>>>>>>> +++ b/hw/block/vhost-user-blk.c
>>>>>>>> @@ -134,10 +134,7 @@ static bool 
>>>>>>>> vhost_user_blk_inflight_needed(void *opaque)
>>>>>>>>   {
>>>>>>>>       struct VHostUserBlk *s = opaque;
>>>>>>>>   -    bool inflight_migration = virtio_has_feature(s- 
>>>>>>>> >dev.protocol_features,
>>>>>>>> - VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
>>>>>>>> -
>>>>>>>> -    return inflight_migration;
>>>>>>>> +    return s->inflight_migration;
>>>>>>>>   }
>>>>>>>>     @@ -232,11 +229,14 @@ static int 
>>>>>>>> vhost_user_blk_stop(VirtIODevice *vdev)
>>>>>>>>           return 0;
>>>>>>>>       }
>>>>>>>>   +    bool skip_drain = vhost_user_blk_inflight_needed(s) &&
>>>>>>>
>>>>>>> nit: declarations on top
>>>>>>>
>>>>>>>> + runstate_check(RUN_STATE_FINISH_MIGRATE);
>>>>>>>> +
>>>>>>>>       force_stop = s->skip_get_vring_base_on_force_shutdown &&
>>>>>>>>                    qemu_force_shutdown_requested();
>>>>>>>>         ret = force_stop ? vhost_dev_force_stop(&s->dev, vdev, 
>>>>>>>> true) :
>>>>>>>> -                       vhost_dev_stop(&s->dev, vdev, true, false);
>>>>>>>> +                       vhost_dev_stop(&s->dev, vdev, true, 
>>>>>>>> skip_drain);
>>>>>>>>         if (k->set_guest_notifiers(qbus->parent, s->dev.nvqs, 
>>>>>>>> false) < 0) {
>>>>>>>>           error_report("vhost guest notifier cleanup failed: 
>>>>>>>> %d", ret);
>>>>>>>> @@ -364,7 +364,6 @@ static int 
>>>>>>>> vhost_user_blk_connect(DeviceState *dev, Error **errp)
>>>>>>>>       vhost_dev_set_config_notifier(&s->dev, &blk_ops);
>>>>>>>>         s->vhost_user.supports_config = true;
>>>>>>>> -    s->vhost_user.supports_inflight_migration = s- 
>>>>>>>> >inflight_migration;
>>>>>>>>       ret = vhost_dev_init(&s->dev, &s->vhost_user, 
>>>>>>>> VHOST_BACKEND_TYPE_USER, 0,
>>>>>>>>                            errp);
>>>>>>>>       if (ret < 0) {
>>>>>>>> @@ -580,10 +579,29 @@ static struct vhost_dev 
>>>>>>>> *vhost_user_blk_get_vhost(VirtIODevice *vdev)
>>>>>>>>       return &s->dev;
>>>>>>>>   }
>>>>>>>>   +static bool vhost_user_blk_pre_save(void *opaque, Error **errp)
>>>>>>>> +{
>>>>>>>> +    VHostUserBlk *s = VHOST_USER_BLK(opaque);
>>>>>>>> +
>>>>>>>> +    bool inflight_migration_enabled = virtio_has_feature(
>>>>>>>> +        s->dev.protocol_features,
>>>>>>>> + VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
>>>>>>>> +    if (vhost_user_blk_inflight_needed(s) && ! 
>>>>>>>> inflight_migration_enabled) {
>>>>>>>> +        error_setg(errp, "can't migrate vhost-user-blk device: "
>>>>>>>> +                         "backend doesn't support "
>>>>>>>> + "VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT "
>>>>>>>> +                         "protocol feature");
>>>>>>>> +        return false;
>>>>>>>> +    }
>>>>>>>> +
>>>>>>>> +    return true;
>>>>>>>> +}
>>>>>>>> +
>>>>>>>>   static const VMStateDescription 
>>>>>>>> vmstate_vhost_user_blk_inflight = {
>>>>>>>>       .name = "vhost-user-blk/inflight",
>>>>>>>>       .version_id = 1,
>>>>>>>>       .needed = vhost_user_blk_inflight_needed,
>>>>>>>> +    .pre_save_errp = vhost_user_blk_pre_save,
>>>>>>>>       .fields = (const VMStateField[]) {
>>>>>>>>           VMSTATE_VHOST_INFLIGHT_REGION(inflight, VHostUserBlk),
>>>>>>>>           VMSTATE_END_OF_LIST()
>>>>>>>> diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c
>>>>>>>> index bb8f8eab77..ed95ec7523 100644
>>>>>>>> --- a/hw/virtio/vhost-user.c
>>>>>>>> +++ b/hw/virtio/vhost-user.c
>>>>>>>> @@ -2225,8 +2225,7 @@ static int vhost_user_backend_init(struct 
>>>>>>>> vhost_dev *dev, void *opaque,
>>>>>>>>               }
>>>>>>>>           }
>>>>>>>>   -        if (!u->user->supports_inflight_migration ||
>>>>>>>> -            !virtio_has_feature(protocol_features,
>>>>>>>> +        if (!virtio_has_feature(protocol_features,
>>>>>>>> VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD)) {
>>>>>>>>               protocol_features &= ~(1ULL <<
>>>>>>>> VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
>>>>>>>> diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
>>>>>>>> index ac4d6fca73..54e4ce2660 100644
>>>>>>>> --- a/hw/virtio/vhost.c
>>>>>>>> +++ b/hw/virtio/vhost.c
>>>>>>>> @@ -1398,6 +1398,7 @@ static int do_vhost_virtqueue_stop(struct 
>>>>>>>> vhost_dev *dev,
>>>>>>>>       int vhost_vq_index = dev->vhost_ops- 
>>>>>>>> >vhost_get_vq_index(dev, idx);
>>>>>>>>       struct vhost_vring_state state = {
>>>>>>>>           .index = vhost_vq_index,
>>>>>>>> +        .num = skip_drain,
>>>>>>>>       };
>>>>>>>>       int r = 0;
>>>>>>>>   diff --git a/include/hw/virtio/vhost-user.h b/include/hw/ 
>>>>>>>> virtio/ vhost-user.h
>>>>>>>> index 53fe996686..c95bad5ddc 100644
>>>>>>>> --- a/include/hw/virtio/vhost-user.h
>>>>>>>> +++ b/include/hw/virtio/vhost-user.h
>>>>>>>> @@ -69,7 +69,6 @@ typedef struct VhostUserState {
>>>>>>>>       GPtrArray *notifiers;
>>>>>>>>       int memory_slots;
>>>>>>>>       bool supports_config;
>>>>>>>> -    bool supports_inflight_migration;
>>>>>>>>   } VhostUserState;
>>>>>>>>     /**
>>>>>>>
>>>>>
>>>

Re: [PATCH v3 5/5] vhost-user: add skip_drain param to GET_VRING_BASE

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 3396 bytes --]

On Mon, May 25, 2026 at 05:34:37PM -0400, Raphael Norwitz wrote:
> Apologies for the late reply.
> 
> On 5/12/26 1:55 AM, Alexandr Moshkov wrote:
> > Gentle ping :)
> > 
> 
> [...]
> 
> > > > > > On 4/16/26 5:26 AM, Alexandr Moshkov wrote:
> > > > > > > Greetings! Thanks for reply!
> > > > > > > 
> > > > > > > 
> > > > > > > On 4/15/26 20:21, Raphael Norwitz wrote:
> > > > > > > > I’m not sure I like using the num field in
> > > > > > > > vhost_vring_state to set magic values which
> > > > > > > > affect protocol behavior for GET_VRING_BASE. It
> > > > > > > > feels like a hack to me. I would think the
> > > > > > > > proper solution if we want to support migration
> > > > > > > > from new to old would be to have new use a
> > > > > > > > different new message entirely. Can we do that?
> > > > > > > 
> > > > > > > I think we can, but I thought at first that this
> > > > > > > will be almost a complete copy of GET_VRING_BASE
> > > > > > > message, with the exception of waiting for drain of
> > > > > > > requests, so I choose to expand existing message.
> > > > > > > 
> > > > > > 
> > > > > > I think a new message would be cleaner. Anyone else have thoughts?
> > > > > > 
> > > > > > > If this is not an appropriate approach, is it better
> > > > > > > to make a new message like GET_VRING_BASE or a
> > > > > > > separate message used together with default
> > > > > > > GET_VRING_BASE message (for example, message for
> > > > > > > setting some kind of status on the server)? What
> > > > > > > should I name this new message?
> > > > > > > 
> > > > > > 
> > > > > > Maybe GET_VRING_BASE_SKIP_DRAIN? How would a separate
> > > > > > message used together with default GET_VRING_BASE
> > > > > > message work?
> > > > > 
> > > > > I was thinking about a message something like
> > > > > SET_VRING_ENABLE - for example SET_SKIP_DRAIN_ENABLE, that
> > > > > would enable/disable some state (skip_drain) in the backend.
> > > > 
> > > > I'd need to see the flow in more detail but sounds promising.
> > > 
> > > On migration vhost-user-blk firstly send SET_SKIP_DRAIN_ENABLE with
> > > num = 1 message if `inflight-migration` device parameter
> > > and VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT enabled. Then send
> > > GET_VRING_BASE and continue work as usual.
> > > 
> > > On SET_SKIP_DRAIN_ENABLE backend sets inner state (skip_drain) so
> > > when GET_VRING_BASE is called and skip_drain is true the drain will
> > > skipped.
> > > 
> > > What do you think?
> > > 
> 
> I'm happy with that in theory but would like more clarity on the details. In
> particular, I'm not sure what you mean by "vhost-user-blk firstly send
> SET_SKIP_DRAIN_ENABLE". To add a new message I would think we would have to
> do a vhost-user protocol feature negotiation to gate it.
> 
> Also what are the precise semantics for SET_SKIP_DRAIN_ENABLE? Would it be
> sent on backend connect or when we're about to migrate?
> 
> I was hoping others would comment but at this point I'd suggest drafting the
> code and then we can re-review.

Adding a new GET_VRING_BASE_SKIP_DRAIN message seems simpler to me than
a stateful SET_SKIP_DRAIN_ENABLE where the back-end needs to stash the
enable_skip_drain state and the front-end would have to toggle the state
if it switches between skip drain and classic behavior.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v3 5/5] vhost-user: add skip_drain param to GET_VRING_BASE

[Raphael Norwitz]

On 5/26/26 2:48 PM, Stefan Hajnoczi wrote:
> On Mon, May 25, 2026 at 05:34:37PM -0400, Raphael Norwitz wrote:
>> Apologies for the late reply.
>>
>> On 5/12/26 1:55 AM, Alexandr Moshkov wrote:
>>> Gentle ping :)
>>>
>>
>> [...]
>>
>>>>>>> On 4/16/26 5:26 AM, Alexandr Moshkov wrote:
>>>>>>>> Greetings! Thanks for reply!
>>>>>>>>
>>>>>>>>
>>>>>>>> On 4/15/26 20:21, Raphael Norwitz wrote:
>>>>>>>>> I’m not sure I like using the num field in
>>>>>>>>> vhost_vring_state to set magic values which
>>>>>>>>> affect protocol behavior for GET_VRING_BASE. It
>>>>>>>>> feels like a hack to me. I would think the
>>>>>>>>> proper solution if we want to support migration
>>>>>>>>> from new to old would be to have new use a
>>>>>>>>> different new message entirely. Can we do that?
>>>>>>>>
>>>>>>>> I think we can, but I thought at first that this
>>>>>>>> will be almost a complete copy of GET_VRING_BASE
>>>>>>>> message, with the exception of waiting for drain of
>>>>>>>> requests, so I choose to expand existing message.
>>>>>>>>
>>>>>>>
>>>>>>> I think a new message would be cleaner. Anyone else have thoughts?
>>>>>>>
>>>>>>>> If this is not an appropriate approach, is it better
>>>>>>>> to make a new message like GET_VRING_BASE or a
>>>>>>>> separate message used together with default
>>>>>>>> GET_VRING_BASE message (for example, message for
>>>>>>>> setting some kind of status on the server)? What
>>>>>>>> should I name this new message?
>>>>>>>>
>>>>>>>
>>>>>>> Maybe GET_VRING_BASE_SKIP_DRAIN? How would a separate
>>>>>>> message used together with default GET_VRING_BASE
>>>>>>> message work?
>>>>>>
>>>>>> I was thinking about a message something like
>>>>>> SET_VRING_ENABLE - for example SET_SKIP_DRAIN_ENABLE, that
>>>>>> would enable/disable some state (skip_drain) in the backend.
>>>>>
>>>>> I'd need to see the flow in more detail but sounds promising.
>>>>
>>>> On migration vhost-user-blk firstly send SET_SKIP_DRAIN_ENABLE with
>>>> num = 1 message if `inflight-migration` device parameter
>>>> and VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT enabled. Then send
>>>> GET_VRING_BASE and continue work as usual.
>>>>
>>>> On SET_SKIP_DRAIN_ENABLE backend sets inner state (skip_drain) so
>>>> when GET_VRING_BASE is called and skip_drain is true the drain will
>>>> skipped.
>>>>
>>>> What do you think?
>>>>
>>
>> I'm happy with that in theory but would like more clarity on the details. In
>> particular, I'm not sure what you mean by "vhost-user-blk firstly send
>> SET_SKIP_DRAIN_ENABLE". To add a new message I would think we would have to
>> do a vhost-user protocol feature negotiation to gate it.
>>
>> Also what are the precise semantics for SET_SKIP_DRAIN_ENABLE? Would it be
>> sent on backend connect or when we're about to migrate?
>>
>> I was hoping others would comment but at this point I'd suggest drafting the
>> code and then we can re-review.
> 
> Adding a new GET_VRING_BASE_SKIP_DRAIN message seems simpler to me than
> a stateful SET_SKIP_DRAIN_ENABLE where the back-end needs to stash the
> enable_skip_drain state and the front-end would have to toggle the state
> if it switches between skip drain and classic behavior.

Agreed - that sounds cleaner.


> 
> Stefan