* [PATCH v5 0/6] vhost-user-blk: add compatibility with older qemu versions
@ 2026-07-28 10:08 Alexandr Moshkov
2026-07-28 10:08 ` [PATCH v5 1/6] vhost-user: add skip_drain param to do_vhost_virtqueue_stop Alexandr Moshkov
` (6 more replies)
0 siblings, 7 replies; 18+ messages in thread
From: Alexandr Moshkov @ 2026-07-28 10:08 UTC (permalink / raw)
To: qemu-devel
Cc: Hanna Reitz, Jason Wang, Vladimir Sementsov-Ogievskiy,
Gonglei (Arei), Jason Wang, Fam Zheng, qemu-block,
Stefan Hajnoczi, Alex Bennée, yc-core@yandex-team.ru,
Pierrick Bouvier, Pierrick Bouvier, Michael S. Tsirkin, virtio-fs,
Kevin Wolf, zhenwei pi, Paolo Bonzini, Stefano Garzarella,
Milan Zamazal, Raphael Norwitz, Alexandr Moshkov
v4 -> v5:
- introduce protocol feature VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN
to guard the new message, instead of reusing the existing
VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INLIFGHT. Update docs.
- improved commit messages.
v3 -> v4:
- add new protocol message GET_VRING_BASE_SKIP_DRAIN instead of parameter to GET_VRING_BASE message. This new message can be send instead of GET_VRING_BASE, allowing back-end to suspend inflight I/O immediately instead of waiting and completing them.
- rebase to newer master
v2 -> v3:
- fix complile problems
- add assert check in do_vhost_virtio_stop
- make inflight-migration property mutable
v1 -> v2:
- reorganize commits: make refactor commits first, then core semantic change
- add additional pre_save check for inflight migration possibility
---
This is v5 of the series previously sent as
"[PATCH v4 0/6] vhost-user-blk: fix inflight migration compatibility".
This series extends the vhost-user-blk inflight migration feature
introduced in QEMU 11.0 to address a runtime compatibility problem.
Currently, the inflight migration behaviour in vhost-user-blk is
controlled by VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT, which is
negotiated once at connection time. Once the feature is negotiated, the
back-end always uses suspend semantics on GET_VRING_BASE — there is no
way for the front-end to request normal drain behaviour on a per-call
basis without reconnecting. This makes it impossible to disable
inflight-migration at runtime, which is needed when, for example,
migrating a VM to an older QEMU version that does not support the
feature.
To solve this, the series introduces a new protocol message
VHOST_USER_GET_VRING_BASE_SKIP_DRAIN (id=45) guarded by a new protocol
feature VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN. The message is
semantically identical to GET_VRING_BASE but explicitly instructs the
back-end to suspend in-flight I/O immediately rather than draining it.
This gives the front-end explicit per-call control: GET_VRING_BASE for
normal operation, GET_VRING_BASE_SKIP_DRAIN during live migration.
In vhost-user-blk, GET_VRING_BASE_SKIP_DRAIN is sent only when
inflight-migration is enabled and the device is stopping due to live
migration (RUN_STATE_FINISH_MIGRATE). The inflight-migration property
is also made mutable so it can be toggled via qom-set at runtime without
reconnecting to the back-end.
Alexandr Moshkov (6):
vhost-user: add skip_drain param to do_vhost_virtqueue_stop
vhost-user: add GET_VRING_BASE_SKIP_DRAIN message
vhost-user: use skip_drain with GET_VRING_BASE_SKIP_DRAIN message
vhost-user-blk: make inflight-migration prop mutable
vhost-user-blk: move inflight_needed higher
vhost-user-blk: use GET_VRING_BASE_SKIP_DRAIN during inflight
migration
backends/cryptodev-vhost.c | 2 +-
backends/vhost-user.c | 2 +-
docs/interop/vhost-user.rst | 88 +++++++++++++++++++------------
hw/block/vhost-user-blk.c | 43 ++++++++++++---
hw/net/vhost_net.c | 9 ++--
hw/scsi/vhost-scsi-common.c | 2 +-
hw/virtio/vdpa-dev.c | 2 +-
hw/virtio/vhost-user-base.c | 2 +-
hw/virtio/vhost-user-fs.c | 2 +-
hw/virtio/vhost-user-scmi.c | 2 +-
hw/virtio/vhost-user.c | 47 ++++++++++++++---
hw/virtio/vhost-vsock-common.c | 2 +-
hw/virtio/vhost.c | 34 ++++++++----
include/hw/virtio/vhost-backend.h | 1 +
include/hw/virtio/vhost-user.h | 2 +-
include/hw/virtio/vhost.h | 7 ++-
16 files changed, 172 insertions(+), 75 deletions(-)
--
2.34.1
^ permalink raw reply [flat|nested] 18+ messages in thread* [PATCH v5 1/6] vhost-user: add skip_drain param to do_vhost_virtqueue_stop 2026-07-28 10:08 [PATCH v5 0/6] vhost-user-blk: add compatibility with older qemu versions Alexandr Moshkov @ 2026-07-28 10:08 ` Alexandr Moshkov 2026-07-28 10:08 ` [PATCH v5 2/6] vhost-user: add GET_VRING_BASE_SKIP_DRAIN message Alexandr Moshkov ` (5 subsequent siblings) 6 siblings, 0 replies; 18+ messages in thread From: Alexandr Moshkov @ 2026-07-28 10:08 UTC (permalink / raw) To: qemu-devel Cc: Hanna Reitz, Jason Wang, Vladimir Sementsov-Ogievskiy, Gonglei (Arei), Jason Wang, Fam Zheng, qemu-block, Stefan Hajnoczi, Alex Bennée, yc-core@yandex-team.ru, Pierrick Bouvier, Pierrick Bouvier, Michael S. Tsirkin, virtio-fs, Kevin Wolf, zhenwei pi, Paolo Bonzini, Stefano Garzarella, Milan Zamazal, Raphael Norwitz, Alexandr Moshkov Currently do_vhost_virtqueue_stop always sends GET_VRING_BASE to the back-end, which requires the back-end to drain all in-flight I/O before stopping the vring. Add a skip_drain parameter to do_vhost_virtqueue_stop and propagate it up through vhost_virtqueue_stop, vhost_dev_stop and their callers. The parameter will be used in a follow-up commit to send a new protocol message that instructs the back-end to suspend in-flight I/O immediately instead of draining it. Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru> --- backends/cryptodev-vhost.c | 2 +- backends/vhost-user.c | 2 +- hw/block/vhost-user-blk.c | 2 +- hw/net/vhost_net.c | 9 +++++---- hw/scsi/vhost-scsi-common.c | 2 +- hw/virtio/vdpa-dev.c | 2 +- hw/virtio/vhost-user-base.c | 2 +- hw/virtio/vhost-user-fs.c | 2 +- hw/virtio/vhost-user-scmi.c | 2 +- hw/virtio/vhost-vsock-common.c | 2 +- hw/virtio/vhost.c | 28 +++++++++++++++++++--------- include/hw/virtio/vhost.h | 7 +++++-- 12 files changed, 38 insertions(+), 24 deletions(-) diff --git a/backends/cryptodev-vhost.c b/backends/cryptodev-vhost.c index c6069f4e5b..f1ca6bcd4e 100644 --- a/backends/cryptodev-vhost.c +++ b/backends/cryptodev-vhost.c @@ -109,7 +109,7 @@ static void cryptodev_vhost_stop_one(CryptoDevBackendVhost *crypto, VirtIODevice *dev) { - vhost_dev_stop(&crypto->dev, dev, false); + vhost_dev_stop(&crypto->dev, dev, false, false); vhost_dev_disable_notifiers(&crypto->dev, dev); } diff --git a/backends/vhost-user.c b/backends/vhost-user.c index 380d825023..0423c2d4ec 100644 --- a/backends/vhost-user.c +++ b/backends/vhost-user.c @@ -108,7 +108,7 @@ vhost_user_backend_stop(VhostUserBackend *b) return 0; } - ret = vhost_dev_stop(&b->dev, b->vdev, true); + ret = vhost_dev_stop(&b->dev, b->vdev, true, false); if (k->set_guest_notifiers && (err = k->set_guest_notifiers(qbus->parent, b->dev.nvqs, false)) < 0) { diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c index b2e46eb3f9..e3b873af7c 100644 --- a/hw/block/vhost-user-blk.c +++ b/hw/block/vhost-user-blk.c @@ -232,7 +232,7 @@ static int vhost_user_blk_stop(VirtIODevice *vdev) qemu_force_shutdown_requested(); ret = force_stop ? vhost_dev_force_stop(&s->dev, vdev, true) : - vhost_dev_stop(&s->dev, vdev, true); + vhost_dev_stop(&s->dev, vdev, true, false); err = k->set_guest_notifiers(qbus->parent, s->dev.nvqs, false); if (err < 0) { diff --git a/hw/net/vhost_net.c b/hw/net/vhost_net.c index 323d117735..6e05c995f1 100644 --- a/hw/net/vhost_net.c +++ b/hw/net/vhost_net.c @@ -384,7 +384,7 @@ fail: if (net->nc->info->poll) { net->nc->info->poll(net->nc, true); } - vhost_dev_stop(&net->dev, dev, false); + vhost_dev_stop(&net->dev, dev, false, false); fail_start: return r; } @@ -403,7 +403,7 @@ static void vhost_net_stop_one(struct vhost_net *net, if (net->nc->info->poll) { net->nc->info->poll(net->nc, true); } - vhost_dev_stop(&net->dev, dev, false); + vhost_dev_stop(&net->dev, dev, false, false); if (net->nc->info->stop) { net->nc->info->stop(net->nc); } @@ -636,7 +636,8 @@ void vhost_net_virtqueue_reset(VirtIODevice *vdev, NetClientState *nc, vhost_virtqueue_stop(&net->dev, vdev, net->dev.vqs + idx, - net->dev.vq_index + idx); + net->dev.vq_index + idx, + false); } int vhost_net_virtqueue_restart(VirtIODevice *vdev, NetClientState *nc, @@ -686,7 +687,7 @@ err_start: assert(ret >= 0); } - vhost_dev_stop(&net->dev, vdev, false); + vhost_dev_stop(&net->dev, vdev, false, false); return r; } diff --git a/hw/scsi/vhost-scsi-common.c b/hw/scsi/vhost-scsi-common.c index e19800a0bc..e546a6dc75 100644 --- a/hw/scsi/vhost-scsi-common.c +++ b/hw/scsi/vhost-scsi-common.c @@ -108,7 +108,7 @@ int vhost_scsi_common_stop(VHostSCSICommon *vsc) VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(qbus); int ret = 0; - ret = vhost_dev_stop(&vsc->dev, vdev, true); + ret = vhost_dev_stop(&vsc->dev, vdev, true, false); if (k->set_guest_notifiers) { int r = k->set_guest_notifiers(qbus->parent, vsc->dev.nvqs, false); diff --git a/hw/virtio/vdpa-dev.c b/hw/virtio/vdpa-dev.c index 94188d37bb..b86d0b664a 100644 --- a/hw/virtio/vdpa-dev.c +++ b/hw/virtio/vdpa-dev.c @@ -300,7 +300,7 @@ static void vhost_vdpa_device_stop(VirtIODevice *vdev) return; } - vhost_dev_stop(&s->dev, vdev, false); + vhost_dev_stop(&s->dev, vdev, false, false); ret = k->set_guest_notifiers(qbus->parent, s->dev.nvqs, false); if (ret < 0) { diff --git a/hw/virtio/vhost-user-base.c b/hw/virtio/vhost-user-base.c index 39b5e637fc..abeae1a064 100644 --- a/hw/virtio/vhost-user-base.c +++ b/hw/virtio/vhost-user-base.c @@ -78,7 +78,7 @@ static int vub_stop(VirtIODevice *vdev) return 0; } - ret = vhost_dev_stop(&vub->vhost_dev, vdev, true); + ret = vhost_dev_stop(&vub->vhost_dev, vdev, true, false); err = k->set_guest_notifiers(qbus->parent, vub->vhost_dev.nvqs, false); if (err < 0) { diff --git a/hw/virtio/vhost-user-fs.c b/hw/virtio/vhost-user-fs.c index 209993918a..0d8842817a 100644 --- a/hw/virtio/vhost-user-fs.c +++ b/hw/virtio/vhost-user-fs.c @@ -111,7 +111,7 @@ static int vuf_stop(VirtIODevice *vdev) return 0; } - ret = vhost_dev_stop(&fs->vhost_dev, vdev, true); + ret = vhost_dev_stop(&fs->vhost_dev, vdev, true, false); err = k->set_guest_notifiers(qbus->parent, fs->vhost_dev.nvqs, false); if (err < 0) { diff --git a/hw/virtio/vhost-user-scmi.c b/hw/virtio/vhost-user-scmi.c index 9470f68c1f..9fa65d7d8b 100644 --- a/hw/virtio/vhost-user-scmi.c +++ b/hw/virtio/vhost-user-scmi.c @@ -101,7 +101,7 @@ static int vu_scmi_stop(VirtIODevice *vdev) return 0; } - ret = vhost_dev_stop(vhost_dev, vdev, true); + ret = vhost_dev_stop(vhost_dev, vdev, true, false); err = k->set_guest_notifiers(qbus->parent, vhost_dev->nvqs, false); if (err < 0) { diff --git a/hw/virtio/vhost-vsock-common.c b/hw/virtio/vhost-vsock-common.c index b79f4c9ce6..4ef037627b 100644 --- a/hw/virtio/vhost-vsock-common.c +++ b/hw/virtio/vhost-vsock-common.c @@ -106,7 +106,7 @@ int vhost_vsock_common_stop(VirtIODevice *vdev) return 0; } - ret = vhost_dev_stop(&vvc->vhost_dev, vdev, true); + ret = vhost_dev_stop(&vvc->vhost_dev, vdev, true, false); err = k->set_guest_notifiers(qbus->parent, vvc->vhost_dev.nvqs, false); if (err < 0) { diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c index af41841b52..2a62550222 100644 --- a/hw/virtio/vhost.c +++ b/hw/virtio/vhost.c @@ -23,6 +23,7 @@ #include "qemu/log.h" #include "standard-headers/linux/vhost_types.h" #include "hw/virtio/virtio-bus.h" +#include "hw/virtio/vhost-user.h" #include "hw/mem/memory-device.h" #include "migration/blocker.h" #include "migration/qemu-file-types.h" @@ -1496,8 +1497,13 @@ fail: static int do_vhost_virtqueue_stop(struct vhost_dev *dev, struct VirtIODevice *vdev, struct vhost_virtqueue *vq, - unsigned idx, bool force) + unsigned idx, bool force, + bool skip_drain) { + if (skip_drain) { + assert(vhost_user_has_protocol_feature(dev, + VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT)); + } int vhost_vq_index = dev->vhost_ops->vhost_get_vq_index(dev, idx); struct vhost_vring_state state = { .index = vhost_vq_index, @@ -1547,9 +1553,10 @@ static int do_vhost_virtqueue_stop(struct vhost_dev *dev, int vhost_virtqueue_stop(struct vhost_dev *dev, struct VirtIODevice *vdev, struct vhost_virtqueue *vq, - unsigned idx) + unsigned idx, + bool skip_drain) { - return do_vhost_virtqueue_stop(dev, vdev, vq, idx, false); + return do_vhost_virtqueue_stop(dev, vdev, vq, idx, false, skip_drain); } static int vhost_virtqueue_set_busyloop_timeout(struct vhost_dev *dev, @@ -2302,7 +2309,8 @@ fail_vq: vhost_virtqueue_stop(hdev, vdev, hdev->vqs + i, - hdev->vq_index + i); + hdev->vq_index + i, + false); } fail_mem: @@ -2317,7 +2325,7 @@ fail_features: /* Host notifiers must be enabled at this point. */ static int do_vhost_dev_stop(struct vhost_dev *hdev, VirtIODevice *vdev, - bool vrings, bool force) + bool vrings, bool force, bool skip_drain) { int i; int rc = 0; @@ -2344,7 +2352,8 @@ static int do_vhost_dev_stop(struct vhost_dev *hdev, VirtIODevice *vdev, vdev, hdev->vqs + i, hdev->vq_index + i, - force); + force, + skip_drain); } if (hdev->vhost_ops->vhost_reset_status) { hdev->vhost_ops->vhost_reset_status(hdev); @@ -2366,15 +2375,16 @@ static int do_vhost_dev_stop(struct vhost_dev *hdev, VirtIODevice *vdev, return rc; } -int vhost_dev_stop(struct vhost_dev *hdev, VirtIODevice *vdev, bool vrings) +int vhost_dev_stop(struct vhost_dev *hdev, VirtIODevice *vdev, bool vrings, + bool skip_drain) { - return do_vhost_dev_stop(hdev, vdev, vrings, false); + return do_vhost_dev_stop(hdev, vdev, vrings, false, skip_drain); } int vhost_dev_force_stop(struct vhost_dev *hdev, VirtIODevice *vdev, bool vrings) { - return do_vhost_dev_stop(hdev, vdev, vrings, true); + return do_vhost_dev_stop(hdev, vdev, vrings, true, false); } int vhost_net_set_backend(struct vhost_dev *hdev, diff --git a/include/hw/virtio/vhost.h b/include/hw/virtio/vhost.h index 684bafcaad..33b88f95ea 100644 --- a/include/hw/virtio/vhost.h +++ b/include/hw/virtio/vhost.h @@ -228,6 +228,7 @@ int vhost_dev_start(struct vhost_dev *hdev, VirtIODevice *vdev, bool vrings); * @hdev: common vhost_dev structure * @vdev: the VirtIODevice structure * @vrings: true to have vrings disabled in this call + * @skip_drain: true to notice back-end to skip draining all in-flight requests * * Stop the vhost device. After the device is stopped the notifiers * can be disabled (@vhost_dev_disable_notifiers) and the device can @@ -235,7 +236,8 @@ int vhost_dev_start(struct vhost_dev *hdev, VirtIODevice *vdev, bool vrings); * * Return: 0 on success, != 0 on error when stopping dev. */ -int vhost_dev_stop(struct vhost_dev *hdev, VirtIODevice *vdev, bool vrings); +int vhost_dev_stop(struct vhost_dev *hdev, VirtIODevice *vdev, bool vrings, + bool skip_drain); /** * vhost_dev_force_stop() - force stop the vhost device @@ -393,7 +395,8 @@ int vhost_device_iotlb_miss(struct vhost_dev *dev, uint64_t iova, int write); int vhost_virtqueue_start(struct vhost_dev *dev, struct VirtIODevice *vdev, struct vhost_virtqueue *vq, unsigned idx); int vhost_virtqueue_stop(struct vhost_dev *dev, struct VirtIODevice *vdev, - struct vhost_virtqueue *vq, unsigned idx); + struct vhost_virtqueue *vq, unsigned idx, + bool skip_drain); void vhost_dev_reset_inflight(struct vhost_inflight *inflight); void vhost_dev_free_inflight(struct vhost_inflight *inflight); -- 2.34.1 ^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH v5 2/6] vhost-user: add GET_VRING_BASE_SKIP_DRAIN message 2026-07-28 10:08 [PATCH v5 0/6] vhost-user-blk: add compatibility with older qemu versions Alexandr Moshkov 2026-07-28 10:08 ` [PATCH v5 1/6] vhost-user: add skip_drain param to do_vhost_virtqueue_stop Alexandr Moshkov @ 2026-07-28 10:08 ` Alexandr Moshkov 2026-07-29 9:14 ` Michael S. Tsirkin 2026-07-28 10:08 ` [PATCH v5 3/6] vhost-user: use skip_drain with " Alexandr Moshkov ` (4 subsequent siblings) 6 siblings, 1 reply; 18+ messages in thread From: Alexandr Moshkov @ 2026-07-28 10:08 UTC (permalink / raw) To: qemu-devel Cc: Hanna Reitz, Jason Wang, Vladimir Sementsov-Ogievskiy, Gonglei (Arei), Jason Wang, Fam Zheng, qemu-block, Stefan Hajnoczi, Alex Bennée, yc-core@yandex-team.ru, Pierrick Bouvier, Pierrick Bouvier, Michael S. Tsirkin, virtio-fs, Kevin Wolf, zhenwei pi, Paolo Bonzini, Stefano Garzarella, Milan Zamazal, Raphael Norwitz, Alexandr Moshkov VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT was introduced to allow the back-end to suspend in-flight I/O during GET_VRING_BASE instead of draining it, enabling live migration of in-flight requests. However, this behaviour is tied to the protocol feature itself — once negotiated, there is no way for the front-end to tell the back-end to fall back to the normal drain behaviour on a per-stop basis. This makes it impossible to selectively use suspend semantics only during live migration while keeping drain semantics in other cases. Introduce a separate message VHOST_USER_GET_VRING_BASE_SKIP_DRAIN (id=45) guarded by a new protocol feature VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN. The message is identical to GET_VRING_BASE except that the back-end must immediately suspend all in-flight I/O and record it in the inflight region. This way the front-end has explicit per-call control: send GET_VRING_BASE for normal drain, send GET_VRING_BASE_SKIP_DRAIN when immediate suspend is needed. The new feature requires both VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT and VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD to be negotiated. Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru> --- docs/interop/vhost-user.rst | 88 +++++++++++++++++++------------ hw/virtio/vhost-user.c | 44 ++++++++++++++-- hw/virtio/vhost.c | 2 +- include/hw/virtio/vhost-backend.h | 1 + include/hw/virtio/vhost-user.h | 1 + 5 files changed, 95 insertions(+), 41 deletions(-) diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst index c83ae2accb..d2e0efaeaa 100644 --- a/docs/interop/vhost-user.rst +++ b/docs/interop/vhost-user.rst @@ -445,6 +445,7 @@ replies, except for the following requests: * ``VHOST_USER_GET_FEATURES`` * ``VHOST_USER_GET_PROTOCOL_FEATURES`` * ``VHOST_USER_GET_VRING_BASE`` +* ``VHOST_USER_GET_VRING_BASE_SKIP_DRAIN`` * ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``) * ``VHOST_USER_GET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``) @@ -523,7 +524,7 @@ must start a ring upon receiving a kick (that is, detecting that file descriptor is readable) on the descriptor specified by ``VHOST_USER_SET_VRING_KICK`` or receiving the in-band message ``VHOST_USER_VRING_KICK`` if negotiated, and stop a ring upon receiving -``VHOST_USER_GET_VRING_BASE``. +``VHOST_USER_GET_VRING_BASE`` or ``VHOST_USER_GET_VRING_BASE_SKIP_DRAIN``. Rings can be enabled or disabled by ``VHOST_USER_SET_VRING_ENABLE``. @@ -1101,29 +1102,30 @@ Protocol features .. code:: c - #define VHOST_USER_PROTOCOL_F_MQ 0 - #define VHOST_USER_PROTOCOL_F_LOG_SHMFD 1 - #define VHOST_USER_PROTOCOL_F_RARP 2 - #define VHOST_USER_PROTOCOL_F_REPLY_ACK 3 - #define VHOST_USER_PROTOCOL_F_MTU 4 - #define VHOST_USER_PROTOCOL_F_BACKEND_REQ 5 - #define VHOST_USER_PROTOCOL_F_CROSS_ENDIAN 6 - #define VHOST_USER_PROTOCOL_F_CRYPTO_SESSION 7 - #define VHOST_USER_PROTOCOL_F_PAGEFAULT 8 - #define VHOST_USER_PROTOCOL_F_CONFIG 9 - #define VHOST_USER_PROTOCOL_F_BACKEND_SEND_FD 10 - #define VHOST_USER_PROTOCOL_F_HOST_NOTIFIER 11 - #define VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD 12 - #define VHOST_USER_PROTOCOL_F_RESET_DEVICE 13 - #define VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS 14 - #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS 15 - #define VHOST_USER_PROTOCOL_F_STATUS 16 - #define VHOST_USER_PROTOCOL_F_XEN_MMAP 17 - #define VHOST_USER_PROTOCOL_F_SHARED_OBJECT 18 - #define VHOST_USER_PROTOCOL_F_DEVICE_STATE 19 - #define VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT 20 - #define VHOST_USER_PROTOCOL_F_GPA_ADDRESSES 21 - #define VHOST_USER_PROTOCOL_F_SHMEM_MAP 22 + #define VHOST_USER_PROTOCOL_F_MQ 0 + #define VHOST_USER_PROTOCOL_F_LOG_SHMFD 1 + #define VHOST_USER_PROTOCOL_F_RARP 2 + #define VHOST_USER_PROTOCOL_F_REPLY_ACK 3 + #define VHOST_USER_PROTOCOL_F_MTU 4 + #define VHOST_USER_PROTOCOL_F_BACKEND_REQ 5 + #define VHOST_USER_PROTOCOL_F_CROSS_ENDIAN 6 + #define VHOST_USER_PROTOCOL_F_CRYPTO_SESSION 7 + #define VHOST_USER_PROTOCOL_F_PAGEFAULT 8 + #define VHOST_USER_PROTOCOL_F_CONFIG 9 + #define VHOST_USER_PROTOCOL_F_BACKEND_SEND_FD 10 + #define VHOST_USER_PROTOCOL_F_HOST_NOTIFIER 11 + #define VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD 12 + #define VHOST_USER_PROTOCOL_F_RESET_DEVICE 13 + #define VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS 14 + #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS 15 + #define VHOST_USER_PROTOCOL_F_STATUS 16 + #define VHOST_USER_PROTOCOL_F_XEN_MMAP 17 + #define VHOST_USER_PROTOCOL_F_SHARED_OBJECT 18 + #define VHOST_USER_PROTOCOL_F_DEVICE_STATE 19 + #define VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT 20 + #define VHOST_USER_PROTOCOL_F_GPA_ADDRESSES 21 + #define VHOST_USER_PROTOCOL_F_SHMEM_MAP 22 + #define VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN 23 Front-end message types ----------------------- @@ -1320,17 +1322,11 @@ Front-end message types set to 0. By default, the back-end must complete all inflight I/O requests for the - specified vring before stopping it. - - If the ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT`` protocol - feature has been negotiated, the back-end may suspend in-flight I/O - requests and record them as described in :ref:`Inflight I/O tracking - <inflight_io_tracking>` instead of completing them before stopping the vring. - How to suspend an in-flight request depends on the implementation of the back-end - but it typically can be done by aborting or cancelling the underlying I/O - request. The ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT`` - protocol feature must only be negotiated if - ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD`` is also negotiated. + specified vring before stopping it. If the + ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN`` protocol feature has + been negotiated, the front-end may instead use + ``VHOST_USER_GET_VRING_BASE_SKIP_DRAIN`` to request the back-end to + suspend in-flight I/O immediately. ``VHOST_USER_SET_VRING_KICK`` :id: 12 @@ -1833,6 +1829,28 @@ Front-end message types * The size may be 0 if the region is unused. +``VHOST_USER_GET_VRING_BASE_SKIP_DRAIN`` + :id: 45 + :equivalent ioctl: N/A + :request payload: vring state description + :reply payload: vring descriptor index/indices + + This message requires the ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN`` + protocol feature to be negotiated. + + Identical to ``VHOST_USER_GET_VRING_BASE`` except that the back-end + must not wait for inflight I/O requests to complete before stopping + the vring. Instead, the back-end must immediately suspend all + in-flight I/O requests and record them as described in + :ref:`Inflight I/O tracking <inflight_io_tracking>`. How to suspend + an in-flight request depends on the implementation of the back-end, + but it typically can be done by aborting or cancelling the underlying + I/O request. + + The ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN`` protocol feature + must only be negotiated if both ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT`` + and ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD`` are also negotiated. + Back-end message types ---------------------- diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index d627351f45..bf455daa44 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -101,6 +101,7 @@ typedef enum VhostUserRequest { VHOST_USER_SET_DEVICE_STATE_FD = 42, VHOST_USER_CHECK_DEVICE_STATE = 43, VHOST_USER_GET_SHMEM_CONFIG = 44, + VHOST_USER_GET_VRING_BASE_SKIP_DRAIN = 45, VHOST_USER_MAX } VhostUserRequest; @@ -167,6 +168,7 @@ static const char *vhost_req_name(VhostUserRequest req) VHOST_USER_CASE(GET_SHARED_OBJECT) VHOST_USER_CASE(SET_DEVICE_STATE_FD) VHOST_USER_CASE(CHECK_DEVICE_STATE) + VHOST_USER_CASE(GET_VRING_BASE_SKIP_DRAIN) default: return "<unknown>"; } @@ -1399,12 +1401,18 @@ static VhostUserHostNotifier *fetch_notifier(VhostUserState *u, return g_ptr_array_index(u->notifiers, idx); } -static int vhost_user_get_vring_base(struct vhost_dev *dev, - struct vhost_vring_state *ring) +static int get_vring_base(struct vhost_dev *dev, + struct vhost_vring_state *ring, + bool skip_drain) { int ret; + int request = VHOST_USER_GET_VRING_BASE; + if (skip_drain) { + request = VHOST_USER_GET_VRING_BASE_SKIP_DRAIN; + } + VhostUserMsg msg = { - .hdr.request = VHOST_USER_GET_VRING_BASE, + .hdr.request = request, .hdr.flags = VHOST_USER_VERSION, .payload.state = *ring, .hdr.size = sizeof(msg.payload.state), @@ -1424,9 +1432,9 @@ static int vhost_user_get_vring_base(struct vhost_dev *dev, return ret; } - if (msg.hdr.request != VHOST_USER_GET_VRING_BASE) { + if (msg.hdr.request != request) { error_report("Received unexpected msg type. Expected %d received %d", - VHOST_USER_GET_VRING_BASE, msg.hdr.request); + request, msg.hdr.request); return -EPROTO; } @@ -1440,6 +1448,25 @@ static int vhost_user_get_vring_base(struct vhost_dev *dev, return 0; } +static int vhost_user_get_vring_base(struct vhost_dev *dev, + struct vhost_vring_state *ring) +{ + return get_vring_base(dev, ring, false); +} + +static int vhost_user_get_vring_base_skip_drain(struct vhost_dev *dev, + struct vhost_vring_state *ring) +{ + bool skip_drain_supported = vhost_user_has_protocol_feature(dev, + VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN); + + if (!skip_drain_supported) { + return 0; + } + + return get_vring_base(dev, ring, true); +} + static int vhost_set_vring_file(struct vhost_dev *dev, VhostUserRequest request, struct vhost_vring_file *file) @@ -2551,6 +2578,12 @@ static int vhost_user_backend_init(struct vhost_dev *dev, void *opaque, VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT); } + if (!virtio_has_feature(protocol_features, + VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT)) { + protocol_features &= ~(1ULL << + VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN); + } + /* final set of protocol features */ u->protocol_features = protocol_features; err = vhost_user_set_protocol_features(dev, u->protocol_features); @@ -3409,6 +3442,7 @@ const VhostOps user_ops = { .vhost_set_vring_num = vhost_user_set_vring_num, .vhost_set_vring_base = vhost_user_set_vring_base, .vhost_get_vring_base = vhost_user_get_vring_base, + .vhost_get_vring_base_skip_drain = vhost_user_get_vring_base_skip_drain, .vhost_set_vring_kick = vhost_user_set_vring_kick, .vhost_set_vring_call = vhost_user_set_vring_call, .vhost_set_vring_err = vhost_user_set_vring_err, diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c index 2a62550222..2c7f464d55 100644 --- a/hw/virtio/vhost.c +++ b/hw/virtio/vhost.c @@ -1502,7 +1502,7 @@ static int do_vhost_virtqueue_stop(struct vhost_dev *dev, { if (skip_drain) { assert(vhost_user_has_protocol_feature(dev, - VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT)); + VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN)); } int vhost_vq_index = dev->vhost_ops->vhost_get_vq_index(dev, idx); struct vhost_vring_state state = { diff --git a/include/hw/virtio/vhost-backend.h b/include/hw/virtio/vhost-backend.h index d878d7b733..daa979a1aa 100644 --- a/include/hw/virtio/vhost-backend.h +++ b/include/hw/virtio/vhost-backend.h @@ -190,6 +190,7 @@ typedef struct VhostOps { vhost_set_vring_num_op vhost_set_vring_num; vhost_set_vring_base_op vhost_set_vring_base; vhost_get_vring_base_op vhost_get_vring_base; + vhost_get_vring_base_op vhost_get_vring_base_skip_drain; vhost_set_vring_kick_op vhost_set_vring_kick; vhost_set_vring_call_op vhost_set_vring_call; vhost_set_vring_err_op vhost_set_vring_err; diff --git a/include/hw/virtio/vhost-user.h b/include/hw/virtio/vhost-user.h index 06c360af18..47c13f8677 100644 --- a/include/hw/virtio/vhost-user.h +++ b/include/hw/virtio/vhost-user.h @@ -36,6 +36,7 @@ enum VhostUserProtocolFeature { VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT = 20, VHOST_USER_PROTOCOL_F_GPA_ADDRESSES = 21, VHOST_USER_PROTOCOL_F_SHMEM = 22, + VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN = 23, VHOST_USER_PROTOCOL_F_MAX }; -- 2.34.1 ^ permalink raw reply related [flat|nested] 18+ messages in thread
* Re: [PATCH v5 2/6] vhost-user: add GET_VRING_BASE_SKIP_DRAIN message 2026-07-28 10:08 ` [PATCH v5 2/6] vhost-user: add GET_VRING_BASE_SKIP_DRAIN message Alexandr Moshkov @ 2026-07-29 9:14 ` Michael S. Tsirkin 0 siblings, 0 replies; 18+ messages in thread From: Michael S. Tsirkin @ 2026-07-29 9:14 UTC (permalink / raw) To: Alexandr Moshkov Cc: qemu-devel, Hanna Reitz, Jason Wang, Vladimir Sementsov-Ogievskiy, Gonglei (Arei), Jason Wang, Fam Zheng, qemu-block, Stefan Hajnoczi, Alex Bennée, yc-core@yandex-team.ru, Pierrick Bouvier, Pierrick Bouvier, virtio-fs, Kevin Wolf, zhenwei pi, Paolo Bonzini, Stefano Garzarella, Milan Zamazal, Raphael Norwitz On Tue, Jul 28, 2026 at 03:08:37PM +0500, Alexandr Moshkov wrote: > VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT was introduced to allow > the back-end to suspend in-flight I/O during GET_VRING_BASE instead of > draining it, enabling live migration of in-flight requests. However, > this behaviour is tied to the protocol feature itself — once negotiated, Can we please avoid em-dashes and all kind of unicode fanciness in commit log? I worry if we do this the next step is emojis) Let's stick to ASCII with the exception of UTF-8 in contributor names. Thanks. > there is no way for the front-end to tell the back-end to fall back to > the normal drain behaviour on a per-stop basis. This makes it impossible > to selectively use suspend semantics only during live migration while > keeping drain semantics in other cases. So what are the "other cases"? > > Introduce a separate message VHOST_USER_GET_VRING_BASE_SKIP_DRAIN (id=45) > guarded by a new protocol feature > VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN. The message is identical > to GET_VRING_BASE except that the back-end must immediately suspend all > in-flight I/O and record it in the inflight region. This way the front-end > has explicit per-call control: send GET_VRING_BASE for normal drain, send > GET_VRING_BASE_SKIP_DRAIN when immediate suspend is needed. > > The new feature requires both VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT > and VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD to be negotiated. > > Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru> > --- > docs/interop/vhost-user.rst | 88 +++++++++++++++++++------------ > hw/virtio/vhost-user.c | 44 ++++++++++++++-- > hw/virtio/vhost.c | 2 +- > include/hw/virtio/vhost-backend.h | 1 + > include/hw/virtio/vhost-user.h | 1 + > 5 files changed, 95 insertions(+), 41 deletions(-) > > diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst > index c83ae2accb..d2e0efaeaa 100644 > --- a/docs/interop/vhost-user.rst > +++ b/docs/interop/vhost-user.rst > @@ -445,6 +445,7 @@ replies, except for the following requests: > * ``VHOST_USER_GET_FEATURES`` > * ``VHOST_USER_GET_PROTOCOL_FEATURES`` > * ``VHOST_USER_GET_VRING_BASE`` > +* ``VHOST_USER_GET_VRING_BASE_SKIP_DRAIN`` > * ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``) > * ``VHOST_USER_GET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``) > > @@ -523,7 +524,7 @@ must start a ring upon receiving a kick (that is, detecting that file > descriptor is readable) on the descriptor specified by > ``VHOST_USER_SET_VRING_KICK`` or receiving the in-band message > ``VHOST_USER_VRING_KICK`` if negotiated, and stop a ring upon receiving > -``VHOST_USER_GET_VRING_BASE``. > +``VHOST_USER_GET_VRING_BASE`` or ``VHOST_USER_GET_VRING_BASE_SKIP_DRAIN``. > > Rings can be enabled or disabled by ``VHOST_USER_SET_VRING_ENABLE``. > > @@ -1101,29 +1102,30 @@ Protocol features > > .. code:: c > > - #define VHOST_USER_PROTOCOL_F_MQ 0 > - #define VHOST_USER_PROTOCOL_F_LOG_SHMFD 1 > - #define VHOST_USER_PROTOCOL_F_RARP 2 > - #define VHOST_USER_PROTOCOL_F_REPLY_ACK 3 > - #define VHOST_USER_PROTOCOL_F_MTU 4 > - #define VHOST_USER_PROTOCOL_F_BACKEND_REQ 5 > - #define VHOST_USER_PROTOCOL_F_CROSS_ENDIAN 6 > - #define VHOST_USER_PROTOCOL_F_CRYPTO_SESSION 7 > - #define VHOST_USER_PROTOCOL_F_PAGEFAULT 8 > - #define VHOST_USER_PROTOCOL_F_CONFIG 9 > - #define VHOST_USER_PROTOCOL_F_BACKEND_SEND_FD 10 > - #define VHOST_USER_PROTOCOL_F_HOST_NOTIFIER 11 > - #define VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD 12 > - #define VHOST_USER_PROTOCOL_F_RESET_DEVICE 13 > - #define VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS 14 > - #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS 15 > - #define VHOST_USER_PROTOCOL_F_STATUS 16 > - #define VHOST_USER_PROTOCOL_F_XEN_MMAP 17 > - #define VHOST_USER_PROTOCOL_F_SHARED_OBJECT 18 > - #define VHOST_USER_PROTOCOL_F_DEVICE_STATE 19 > - #define VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT 20 > - #define VHOST_USER_PROTOCOL_F_GPA_ADDRESSES 21 > - #define VHOST_USER_PROTOCOL_F_SHMEM_MAP 22 > + #define VHOST_USER_PROTOCOL_F_MQ 0 > + #define VHOST_USER_PROTOCOL_F_LOG_SHMFD 1 > + #define VHOST_USER_PROTOCOL_F_RARP 2 > + #define VHOST_USER_PROTOCOL_F_REPLY_ACK 3 > + #define VHOST_USER_PROTOCOL_F_MTU 4 > + #define VHOST_USER_PROTOCOL_F_BACKEND_REQ 5 > + #define VHOST_USER_PROTOCOL_F_CROSS_ENDIAN 6 > + #define VHOST_USER_PROTOCOL_F_CRYPTO_SESSION 7 > + #define VHOST_USER_PROTOCOL_F_PAGEFAULT 8 > + #define VHOST_USER_PROTOCOL_F_CONFIG 9 > + #define VHOST_USER_PROTOCOL_F_BACKEND_SEND_FD 10 > + #define VHOST_USER_PROTOCOL_F_HOST_NOTIFIER 11 > + #define VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD 12 > + #define VHOST_USER_PROTOCOL_F_RESET_DEVICE 13 > + #define VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS 14 > + #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS 15 > + #define VHOST_USER_PROTOCOL_F_STATUS 16 > + #define VHOST_USER_PROTOCOL_F_XEN_MMAP 17 > + #define VHOST_USER_PROTOCOL_F_SHARED_OBJECT 18 > + #define VHOST_USER_PROTOCOL_F_DEVICE_STATE 19 > + #define VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT 20 > + #define VHOST_USER_PROTOCOL_F_GPA_ADDRESSES 21 > + #define VHOST_USER_PROTOCOL_F_SHMEM_MAP 22 > + #define VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN 23 > > Front-end message types > ----------------------- > @@ -1320,17 +1322,11 @@ Front-end message types > set to 0. > > By default, the back-end must complete all inflight I/O requests for the > - specified vring before stopping it. > - > - If the ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT`` protocol > - feature has been negotiated, the back-end may suspend in-flight I/O > - requests and record them as described in :ref:`Inflight I/O tracking > - <inflight_io_tracking>` instead of completing them before stopping the vring. > - How to suspend an in-flight request depends on the implementation of the back-end > - but it typically can be done by aborting or cancelling the underlying I/O > - request. The ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT`` > - protocol feature must only be negotiated if > - ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD`` is also negotiated. > + specified vring before stopping it. If the > + ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN`` protocol feature has > + been negotiated, the front-end may instead use > + ``VHOST_USER_GET_VRING_BASE_SKIP_DRAIN`` to request the back-end to > + suspend in-flight I/O immediately. > > ``VHOST_USER_SET_VRING_KICK`` > :id: 12 > @@ -1833,6 +1829,28 @@ Front-end message types > > * The size may be 0 if the region is unused. > > +``VHOST_USER_GET_VRING_BASE_SKIP_DRAIN`` > + :id: 45 > + :equivalent ioctl: N/A > + :request payload: vring state description > + :reply payload: vring descriptor index/indices > + > + This message requires the ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN`` > + protocol feature to be negotiated. > + > + Identical to ``VHOST_USER_GET_VRING_BASE`` except that the back-end > + must not wait for inflight I/O requests to complete before stopping > + the vring. Instead, the back-end must immediately suspend all > + in-flight I/O requests and record them as described in > + :ref:`Inflight I/O tracking <inflight_io_tracking>`. How to suspend > + an in-flight request depends on the implementation of the back-end, > + but it typically can be done by aborting or cancelling the underlying > + I/O request. > + > + The ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN`` protocol feature > + must only be negotiated if both ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT`` > + and ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD`` are also negotiated. > + > Back-end message types > ---------------------- > > diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c > index d627351f45..bf455daa44 100644 > --- a/hw/virtio/vhost-user.c > +++ b/hw/virtio/vhost-user.c > @@ -101,6 +101,7 @@ typedef enum VhostUserRequest { > VHOST_USER_SET_DEVICE_STATE_FD = 42, > VHOST_USER_CHECK_DEVICE_STATE = 43, > VHOST_USER_GET_SHMEM_CONFIG = 44, > + VHOST_USER_GET_VRING_BASE_SKIP_DRAIN = 45, > VHOST_USER_MAX > } VhostUserRequest; > > @@ -167,6 +168,7 @@ static const char *vhost_req_name(VhostUserRequest req) > VHOST_USER_CASE(GET_SHARED_OBJECT) > VHOST_USER_CASE(SET_DEVICE_STATE_FD) > VHOST_USER_CASE(CHECK_DEVICE_STATE) > + VHOST_USER_CASE(GET_VRING_BASE_SKIP_DRAIN) > default: > return "<unknown>"; > } > @@ -1399,12 +1401,18 @@ static VhostUserHostNotifier *fetch_notifier(VhostUserState *u, > return g_ptr_array_index(u->notifiers, idx); > } > > -static int vhost_user_get_vring_base(struct vhost_dev *dev, > - struct vhost_vring_state *ring) > +static int get_vring_base(struct vhost_dev *dev, > + struct vhost_vring_state *ring, > + bool skip_drain) > { > int ret; > + int request = VHOST_USER_GET_VRING_BASE; > + if (skip_drain) { > + request = VHOST_USER_GET_VRING_BASE_SKIP_DRAIN; > + } > + > VhostUserMsg msg = { > - .hdr.request = VHOST_USER_GET_VRING_BASE, > + .hdr.request = request, > .hdr.flags = VHOST_USER_VERSION, > .payload.state = *ring, > .hdr.size = sizeof(msg.payload.state), > @@ -1424,9 +1432,9 @@ static int vhost_user_get_vring_base(struct vhost_dev *dev, > return ret; > } > > - if (msg.hdr.request != VHOST_USER_GET_VRING_BASE) { > + if (msg.hdr.request != request) { > error_report("Received unexpected msg type. Expected %d received %d", > - VHOST_USER_GET_VRING_BASE, msg.hdr.request); > + request, msg.hdr.request); > return -EPROTO; > } > > @@ -1440,6 +1448,25 @@ static int vhost_user_get_vring_base(struct vhost_dev *dev, > return 0; > } > > +static int vhost_user_get_vring_base(struct vhost_dev *dev, > + struct vhost_vring_state *ring) > +{ > + return get_vring_base(dev, ring, false); > +} > + > +static int vhost_user_get_vring_base_skip_drain(struct vhost_dev *dev, > + struct vhost_vring_state *ring) > +{ > + bool skip_drain_supported = vhost_user_has_protocol_feature(dev, > + VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN); > + > + if (!skip_drain_supported) { > + return 0; > + } > + > + return get_vring_base(dev, ring, true); > +} > + > static int vhost_set_vring_file(struct vhost_dev *dev, > VhostUserRequest request, > struct vhost_vring_file *file) > @@ -2551,6 +2578,12 @@ static int vhost_user_backend_init(struct vhost_dev *dev, void *opaque, > VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT); > } > > + if (!virtio_has_feature(protocol_features, > + VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT)) { > + protocol_features &= ~(1ULL << > + VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN); > + } > + > /* final set of protocol features */ > u->protocol_features = protocol_features; > err = vhost_user_set_protocol_features(dev, u->protocol_features); > @@ -3409,6 +3442,7 @@ const VhostOps user_ops = { > .vhost_set_vring_num = vhost_user_set_vring_num, > .vhost_set_vring_base = vhost_user_set_vring_base, > .vhost_get_vring_base = vhost_user_get_vring_base, > + .vhost_get_vring_base_skip_drain = vhost_user_get_vring_base_skip_drain, > .vhost_set_vring_kick = vhost_user_set_vring_kick, > .vhost_set_vring_call = vhost_user_set_vring_call, > .vhost_set_vring_err = vhost_user_set_vring_err, > diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c > index 2a62550222..2c7f464d55 100644 > --- a/hw/virtio/vhost.c > +++ b/hw/virtio/vhost.c > @@ -1502,7 +1502,7 @@ static int do_vhost_virtqueue_stop(struct vhost_dev *dev, > { > if (skip_drain) { > assert(vhost_user_has_protocol_feature(dev, > - VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT)); > + VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN)); > } > int vhost_vq_index = dev->vhost_ops->vhost_get_vq_index(dev, idx); > struct vhost_vring_state state = { > diff --git a/include/hw/virtio/vhost-backend.h b/include/hw/virtio/vhost-backend.h > index d878d7b733..daa979a1aa 100644 > --- a/include/hw/virtio/vhost-backend.h > +++ b/include/hw/virtio/vhost-backend.h > @@ -190,6 +190,7 @@ typedef struct VhostOps { > vhost_set_vring_num_op vhost_set_vring_num; > vhost_set_vring_base_op vhost_set_vring_base; > vhost_get_vring_base_op vhost_get_vring_base; > + vhost_get_vring_base_op vhost_get_vring_base_skip_drain; > vhost_set_vring_kick_op vhost_set_vring_kick; > vhost_set_vring_call_op vhost_set_vring_call; > vhost_set_vring_err_op vhost_set_vring_err; > diff --git a/include/hw/virtio/vhost-user.h b/include/hw/virtio/vhost-user.h > index 06c360af18..47c13f8677 100644 > --- a/include/hw/virtio/vhost-user.h > +++ b/include/hw/virtio/vhost-user.h > @@ -36,6 +36,7 @@ enum VhostUserProtocolFeature { > VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT = 20, > VHOST_USER_PROTOCOL_F_GPA_ADDRESSES = 21, > VHOST_USER_PROTOCOL_F_SHMEM = 22, > + VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN = 23, > VHOST_USER_PROTOCOL_F_MAX > }; > > -- > 2.34.1 ^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH v5 3/6] vhost-user: use skip_drain with GET_VRING_BASE_SKIP_DRAIN message 2026-07-28 10:08 [PATCH v5 0/6] vhost-user-blk: add compatibility with older qemu versions Alexandr Moshkov 2026-07-28 10:08 ` [PATCH v5 1/6] vhost-user: add skip_drain param to do_vhost_virtqueue_stop Alexandr Moshkov 2026-07-28 10:08 ` [PATCH v5 2/6] vhost-user: add GET_VRING_BASE_SKIP_DRAIN message Alexandr Moshkov @ 2026-07-28 10:08 ` Alexandr Moshkov 2026-07-28 10:08 ` [PATCH v5 4/6] vhost-user-blk: make inflight-migration prop mutable Alexandr Moshkov ` (3 subsequent siblings) 6 siblings, 0 replies; 18+ messages in thread From: Alexandr Moshkov @ 2026-07-28 10:08 UTC (permalink / raw) To: qemu-devel Cc: Hanna Reitz, Jason Wang, Vladimir Sementsov-Ogievskiy, Gonglei (Arei), Jason Wang, Fam Zheng, qemu-block, Stefan Hajnoczi, Alex Bennée, yc-core@yandex-team.ru, Pierrick Bouvier, Pierrick Bouvier, Michael S. Tsirkin, virtio-fs, Kevin Wolf, zhenwei pi, Paolo Bonzini, Stefano Garzarella, Milan Zamazal, Raphael Norwitz, Alexandr Moshkov Wire up the skip_drain parameter introduced in the previous commit to the new GET_VRING_BASE_SKIP_DRAIN message. When skip_drain is set, send GET_VRING_BASE_SKIP_DRAIN instead of GET_VRING_BASE, instructing the back-end to suspend in-flight I/O immediately. Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru> --- hw/virtio/vhost.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c index 2c7f464d55..bd079f9080 100644 --- a/hw/virtio/vhost.c +++ b/hw/virtio/vhost.c @@ -1518,7 +1518,11 @@ static int do_vhost_virtqueue_stop(struct vhost_dev *dev, } if (!force) { - r = dev->vhost_ops->vhost_get_vring_base(dev, &state); + if (!skip_drain) { + r = dev->vhost_ops->vhost_get_vring_base(dev, &state); + } else { + r = dev->vhost_ops->vhost_get_vring_base_skip_drain(dev, &state); + } if (r < 0) { VHOST_OPS_DEBUG(r, "vhost VQ %u ring restore failed: %d", idx, r); } -- 2.34.1 ^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH v5 4/6] vhost-user-blk: make inflight-migration prop mutable 2026-07-28 10:08 [PATCH v5 0/6] vhost-user-blk: add compatibility with older qemu versions Alexandr Moshkov ` (2 preceding siblings ...) 2026-07-28 10:08 ` [PATCH v5 3/6] vhost-user: use skip_drain with " Alexandr Moshkov @ 2026-07-28 10:08 ` Alexandr Moshkov 2026-07-29 9:32 ` Michael S. Tsirkin 2026-07-28 10:08 ` [PATCH v5 5/6] vhost-user-blk: move inflight_needed higher Alexandr Moshkov ` (2 subsequent siblings) 6 siblings, 1 reply; 18+ messages in thread From: Alexandr Moshkov @ 2026-07-28 10:08 UTC (permalink / raw) To: qemu-devel Cc: Hanna Reitz, Jason Wang, Vladimir Sementsov-Ogievskiy, Gonglei (Arei), Jason Wang, Fam Zheng, qemu-block, Stefan Hajnoczi, Alex Bennée, yc-core@yandex-team.ru, Pierrick Bouvier, Pierrick Bouvier, Michael S. Tsirkin, virtio-fs, Kevin Wolf, zhenwei pi, Paolo Bonzini, Stefano Garzarella, Milan Zamazal, Raphael Norwitz, Alexandr Moshkov When migrating from a QEMU version that supports inflight-migration to an older one that does not, there is no way to disable the feature at runtime — the VM must be stopped and reconfigured. This is impractical in production environments. Make the inflight-migration property mutable after device realization so it can be toggled via qom-set without restarting the VM. Acked-by: Raphael Norwitz <rnorwitz@nvidia.com> Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru> --- hw/block/vhost-user-blk.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c index e3b873af7c..650c004bde 100644 --- a/hw/block/vhost-user-blk.c +++ b/hw/block/vhost-user-blk.c @@ -619,6 +619,8 @@ static const VMStateDescription vmstate_vhost_user_blk = { } }; +static PropertyInfo vhost_user_blk_inflight_migration_prop; + static const Property vhost_user_blk_properties[] = { DEFINE_PROP_CHR("chardev", VHostUserBlk, chardev), DEFINE_PROP_UINT16("num-queues", VHostUserBlk, num_queues, @@ -632,8 +634,9 @@ static const Property vhost_user_blk_properties[] = { VIRTIO_BLK_F_WRITE_ZEROES, true), DEFINE_PROP_BOOL("skip-get-vring-base-on-force-shutdown", VHostUserBlk, skip_get_vring_base_on_force_shutdown, false), - DEFINE_PROP_BOOL("inflight-migration", VHostUserBlk, - inflight_migration, false), + DEFINE_PROP("inflight-migration", VHostUserBlk, inflight_migration, + vhost_user_blk_inflight_migration_prop, bool, + .set_default = true, .defval.u = false), }; static void vhost_user_blk_class_init(ObjectClass *klass, const void *data) @@ -665,6 +668,9 @@ static const TypeInfo vhost_user_blk_info = { static void virtio_register_types(void) { + vhost_user_blk_inflight_migration_prop = qdev_prop_bool; + vhost_user_blk_inflight_migration_prop.realized_set_allowed = true; + type_register_static(&vhost_user_blk_info); } -- 2.34.1 ^ permalink raw reply related [flat|nested] 18+ messages in thread
* Re: [PATCH v5 4/6] vhost-user-blk: make inflight-migration prop mutable 2026-07-28 10:08 ` [PATCH v5 4/6] vhost-user-blk: make inflight-migration prop mutable Alexandr Moshkov @ 2026-07-29 9:32 ` Michael S. Tsirkin [not found] ` <679a33d1-0f6c-4456-ac76-1f9e1251f2ec@yandex-team.ru> 0 siblings, 1 reply; 18+ messages in thread From: Michael S. Tsirkin @ 2026-07-29 9:32 UTC (permalink / raw) To: Alexandr Moshkov Cc: qemu-devel, Hanna Reitz, Jason Wang, Vladimir Sementsov-Ogievskiy, Gonglei (Arei), Jason Wang, Fam Zheng, qemu-block, Stefan Hajnoczi, Alex Bennée, yc-core@yandex-team.ru, Pierrick Bouvier, Pierrick Bouvier, virtio-fs, Kevin Wolf, zhenwei pi, Paolo Bonzini, Stefano Garzarella, Milan Zamazal, Raphael Norwitz On Tue, Jul 28, 2026 at 03:08:39PM +0500, Alexandr Moshkov wrote: > When migrating from a QEMU version that supports inflight-migration to > an older one that does not, there is no way to disable the feature at > runtime — the VM must be stopped and reconfigured. This is impractical > in production environments. > > Make the inflight-migration property mutable after device realization > so it can be toggled via qom-set without restarting the VM. > > Acked-by: Raphael Norwitz <rnorwitz@nvidia.com> > Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru> I think I am beginning to understand. You are running qemu with inflight-migration on and want to migrate to qemu without inflight-migration at all. Since it is guest transparent you could retrofit it like this. My question is why is it worth it, we do not normally support migrating between qemu versions with different command lines. > --- > hw/block/vhost-user-blk.c | 10 ++++++++-- > 1 file changed, 8 insertions(+), 2 deletions(-) > > diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c > index e3b873af7c..650c004bde 100644 > --- a/hw/block/vhost-user-blk.c > +++ b/hw/block/vhost-user-blk.c > @@ -619,6 +619,8 @@ static const VMStateDescription vmstate_vhost_user_blk = { > } > }; > > +static PropertyInfo vhost_user_blk_inflight_migration_prop; > + > static const Property vhost_user_blk_properties[] = { > DEFINE_PROP_CHR("chardev", VHostUserBlk, chardev), > DEFINE_PROP_UINT16("num-queues", VHostUserBlk, num_queues, > @@ -632,8 +634,9 @@ static const Property vhost_user_blk_properties[] = { > VIRTIO_BLK_F_WRITE_ZEROES, true), > DEFINE_PROP_BOOL("skip-get-vring-base-on-force-shutdown", VHostUserBlk, > skip_get_vring_base_on_force_shutdown, false), > - DEFINE_PROP_BOOL("inflight-migration", VHostUserBlk, > - inflight_migration, false), > + DEFINE_PROP("inflight-migration", VHostUserBlk, inflight_migration, > + vhost_user_blk_inflight_migration_prop, bool, > + .set_default = true, .defval.u = false), > }; > > static void vhost_user_blk_class_init(ObjectClass *klass, const void *data) > @@ -665,6 +668,9 @@ static const TypeInfo vhost_user_blk_info = { > > static void virtio_register_types(void) > { > + vhost_user_blk_inflight_migration_prop = qdev_prop_bool; > + vhost_user_blk_inflight_migration_prop.realized_set_allowed = true; > + > type_register_static(&vhost_user_blk_info); > } So then, for example, let us say I paused the VM, then set the flag, now inflight is on but GET_BASE did not drain it? > -- > 2.34.1 ^ permalink raw reply [flat|nested] 18+ messages in thread
[parent not found: <679a33d1-0f6c-4456-ac76-1f9e1251f2ec@yandex-team.ru>]
* Re: [PATCH v5 4/6] vhost-user-blk: make inflight-migration prop mutable [not found] ` <679a33d1-0f6c-4456-ac76-1f9e1251f2ec@yandex-team.ru> @ 2026-07-29 10:14 ` Michael S. Tsirkin 2026-07-29 10:35 ` Alexandr Moshkov 0 siblings, 1 reply; 18+ messages in thread From: Michael S. Tsirkin @ 2026-07-29 10:14 UTC (permalink / raw) To: Alexandr Moshkov Cc: qemu-devel, Hanna Reitz, Jason Wang, Vladimir Sementsov-Ogievskiy, Gonglei (Arei), Jason Wang, Fam Zheng, qemu-block, Stefan Hajnoczi, Alex Bennée, yc-core@yandex-team.ru, Pierrick Bouvier, Pierrick Bouvier, virtio-fs, Kevin Wolf, zhenwei pi, Paolo Bonzini, Stefano Garzarella, Milan Zamazal, Raphael Norwitz On Wed, Jul 29, 2026 at 03:09:57PM +0500, Alexandr Moshkov wrote: > > On 7/29/26 14:32, Michael S. Tsirkin wrote: > > On Tue, Jul 28, 2026 at 03:08:39PM +0500, Alexandr Moshkov wrote: > > When migrating from a QEMU version that supports inflight-migration to > an older one that does not, there is no way to disable the feature at > runtime — the VM must be stopped and reconfigured. This is impractical > in production environments. > > Make the inflight-migration property mutable after device realization > so it can be toggled via qom-set without restarting the VM. > > Acked-by: Raphael Norwitz <rnorwitz@nvidia.com> > Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru> > > > > I think I am beginning to understand. > > > You are running qemu with inflight-migration on and want to migrate > to qemu without inflight-migration at all. > > > Since it is guest transparent you could retrofit it like this. > > My question is why is it worth it, we do not normally support > migrating between qemu versions with different command lines. > > I think you right. Maybe I've been focusing too much on the ability to migrate > between versions of qemu with and without inflight migration. > > This series allows to turn off and on the inflight-migration feature at runtime > without recreating the VM (it was the only way to turn off the feature, since > the protocol feature could no longer be turned off, after initialization with > the backend). This is more important feature, and it leads to the fact that > this feature allows to migrate between different versions of qemu (with and > without inflight-migration support). > > > > --- > hw/block/vhost-user-blk.c | 10 ++++++++-- > 1 file changed, 8 insertions(+), 2 deletions(-) > > diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c > index e3b873af7c..650c004bde 100644 > --- a/hw/block/vhost-user-blk.c > +++ b/hw/block/vhost-user-blk.c > @@ -619,6 +619,8 @@ static const VMStateDescription vmstate_vhost_user_blk = { > } > }; > > +static PropertyInfo vhost_user_blk_inflight_migration_prop; > + > static const Property vhost_user_blk_properties[] = { > DEFINE_PROP_CHR("chardev", VHostUserBlk, chardev), > DEFINE_PROP_UINT16("num-queues", VHostUserBlk, num_queues, > @@ -632,8 +634,9 @@ static const Property vhost_user_blk_properties[] = { > VIRTIO_BLK_F_WRITE_ZEROES, true), > DEFINE_PROP_BOOL("skip-get-vring-base-on-force-shutdown", VHostUserBlk, > skip_get_vring_base_on_force_shutdown, false), > - DEFINE_PROP_BOOL("inflight-migration", VHostUserBlk, > - inflight_migration, false), > + DEFINE_PROP("inflight-migration", VHostUserBlk, inflight_migration, > + vhost_user_blk_inflight_migration_prop, bool, > + .set_default = true, .defval.u = false), > }; > > static void vhost_user_blk_class_init(ObjectClass *klass, const void *data) > @@ -665,6 +668,9 @@ static const TypeInfo vhost_user_blk_info = { > > static void virtio_register_types(void) > { > + vhost_user_blk_inflight_migration_prop = qdev_prop_bool; > + vhost_user_blk_inflight_migration_prop.realized_set_allowed = true; > + > type_register_static(&vhost_user_blk_info); > } > > So then, for example, let us say I paused the VM, then set the flag, > now inflight is on but GET_BASE did not drain it? > > If I understood the question correctly, this is valid behavior. Before > migration QEMU check protocol features to understand does the backend support > inflight migration. If it does, after that QEMU migrate inflight buffer to > other VM. If it's not, return error before migration started. I apologise i reverted the logic in the question. I start vm and inflight is on. I stop vm get base does not drain. i turn inflight off. now it looks like it will happily migrate? > -- > 2.34.1 > ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v5 4/6] vhost-user-blk: make inflight-migration prop mutable 2026-07-29 10:14 ` Michael S. Tsirkin @ 2026-07-29 10:35 ` Alexandr Moshkov 2026-07-29 10:48 ` Michael S. Tsirkin 0 siblings, 1 reply; 18+ messages in thread From: Alexandr Moshkov @ 2026-07-29 10:35 UTC (permalink / raw) To: Michael S. Tsirkin Cc: qemu-devel, Hanna Reitz, Jason Wang, Vladimir Sementsov-Ogievskiy, Gonglei (Arei), Jason Wang, Fam Zheng, qemu-block, Stefan Hajnoczi, Alex Bennée, yc-core@yandex-team.ru, Pierrick Bouvier, Pierrick Bouvier, virtio-fs, Kevin Wolf, zhenwei pi, Paolo Bonzini, Stefano Garzarella, Milan Zamazal, Raphael Norwitz On 7/29/26 15:14, Michael S. Tsirkin wrote: > On Wed, Jul 29, 2026 at 03:09:57PM +0500, Alexandr Moshkov wrote: >> On 7/29/26 14:32, Michael S. Tsirkin wrote: >> >> On Tue, Jul 28, 2026 at 03:08:39PM +0500, Alexandr Moshkov wrote: >> >> When migrating from a QEMU version that supports inflight-migration to >> an older one that does not, there is no way to disable the feature at >> runtime — the VM must be stopped and reconfigured. This is impractical >> in production environments. >> >> Make the inflight-migration property mutable after device realization >> so it can be toggled via qom-set without restarting the VM. >> >> Acked-by: Raphael Norwitz <rnorwitz@nvidia.com> >> Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru> >> >> >> >> I think I am beginning to understand. >> >> >> You are running qemu with inflight-migration on and want to migrate >> to qemu without inflight-migration at all. >> >> >> Since it is guest transparent you could retrofit it like this. >> >> My question is why is it worth it, we do not normally support >> migrating between qemu versions with different command lines. >> >> I think you right. Maybe I've been focusing too much on the ability to migrate >> between versions of qemu with and without inflight migration. >> >> This series allows to turn off and on the inflight-migration feature at runtime >> without recreating the VM (it was the only way to turn off the feature, since >> the protocol feature could no longer be turned off, after initialization with >> the backend). This is more important feature, and it leads to the fact that >> this feature allows to migrate between different versions of qemu (with and >> without inflight-migration support). >> >> >> >> --- >> hw/block/vhost-user-blk.c | 10 ++++++++-- >> 1 file changed, 8 insertions(+), 2 deletions(-) >> >> diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c >> index e3b873af7c..650c004bde 100644 >> --- a/hw/block/vhost-user-blk.c >> +++ b/hw/block/vhost-user-blk.c >> @@ -619,6 +619,8 @@ static const VMStateDescription vmstate_vhost_user_blk = { >> } >> }; >> >> +static PropertyInfo vhost_user_blk_inflight_migration_prop; >> + >> static const Property vhost_user_blk_properties[] = { >> DEFINE_PROP_CHR("chardev", VHostUserBlk, chardev), >> DEFINE_PROP_UINT16("num-queues", VHostUserBlk, num_queues, >> @@ -632,8 +634,9 @@ static const Property vhost_user_blk_properties[] = { >> VIRTIO_BLK_F_WRITE_ZEROES, true), >> DEFINE_PROP_BOOL("skip-get-vring-base-on-force-shutdown", VHostUserBlk, >> skip_get_vring_base_on_force_shutdown, false), >> - DEFINE_PROP_BOOL("inflight-migration", VHostUserBlk, >> - inflight_migration, false), >> + DEFINE_PROP("inflight-migration", VHostUserBlk, inflight_migration, >> + vhost_user_blk_inflight_migration_prop, bool, >> + .set_default = true, .defval.u = false), >> }; >> >> static void vhost_user_blk_class_init(ObjectClass *klass, const void *data) >> @@ -665,6 +668,9 @@ static const TypeInfo vhost_user_blk_info = { >> >> static void virtio_register_types(void) >> { >> + vhost_user_blk_inflight_migration_prop = qdev_prop_bool; >> + vhost_user_blk_inflight_migration_prop.realized_set_allowed = true; >> + >> type_register_static(&vhost_user_blk_info); >> } >> >> So then, for example, let us say I paused the VM, then set the flag, >> now inflight is on but GET_BASE did not drain it? >> >> If I understood the question correctly, this is valid behavior. Before >> migration QEMU check protocol features to understand does the backend support >> inflight migration. If it does, after that QEMU migrate inflight buffer to >> other VM. If it's not, return error before migration started. > > I apologise i reverted the logic in the question. > > > I start vm and inflight is on. > I stop vm get base does not drain. In case of VM stop, backend wait to drain all requests. Ability to perform drain or not available only during migration by skip_drain variable in vhost_user_blk_stop(). skip_drain true only if inflight_migration is on and runstate is FINISH_MIGRATION. > i turn inflight off. > now it looks like it will happily migrate? So in this case, when vm migrated with inflight off, it will leads to using GET_VRING_BASE message, that wait all requests to be drained. > >> -- >> 2.34.1 >> ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v5 4/6] vhost-user-blk: make inflight-migration prop mutable 2026-07-29 10:35 ` Alexandr Moshkov @ 2026-07-29 10:48 ` Michael S. Tsirkin [not found] ` <dd1f5830-75ff-4256-a179-6a8341a3b4ec@yandex-team.ru> 0 siblings, 1 reply; 18+ messages in thread From: Michael S. Tsirkin @ 2026-07-29 10:48 UTC (permalink / raw) To: Alexandr Moshkov Cc: qemu-devel, Hanna Reitz, Jason Wang, Vladimir Sementsov-Ogievskiy, Gonglei (Arei), Jason Wang, Fam Zheng, qemu-block, Stefan Hajnoczi, Alex Bennée, yc-core@yandex-team.ru, Pierrick Bouvier, Pierrick Bouvier, virtio-fs, Kevin Wolf, zhenwei pi, Paolo Bonzini, Stefano Garzarella, Milan Zamazal, Raphael Norwitz On Wed, Jul 29, 2026 at 03:35:31PM +0500, Alexandr Moshkov wrote: > > On 7/29/26 15:14, Michael S. Tsirkin wrote: > > On Wed, Jul 29, 2026 at 03:09:57PM +0500, Alexandr Moshkov wrote: > > > On 7/29/26 14:32, Michael S. Tsirkin wrote: > > > > > > On Tue, Jul 28, 2026 at 03:08:39PM +0500, Alexandr Moshkov wrote: > > > > > > When migrating from a QEMU version that supports inflight-migration to > > > an older one that does not, there is no way to disable the feature at > > > runtime — the VM must be stopped and reconfigured. This is impractical > > > in production environments. > > > > > > Make the inflight-migration property mutable after device realization > > > so it can be toggled via qom-set without restarting the VM. > > > > > > Acked-by: Raphael Norwitz <rnorwitz@nvidia.com> > > > Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru> > > > > > > > > > > > > I think I am beginning to understand. > > > > > > > > > You are running qemu with inflight-migration on and want to migrate > > > to qemu without inflight-migration at all. > > > > > > > > > Since it is guest transparent you could retrofit it like this. > > > > > > My question is why is it worth it, we do not normally support > > > migrating between qemu versions with different command lines. > > > > > > I think you right. Maybe I've been focusing too much on the ability to migrate > > > between versions of qemu with and without inflight migration. > > > > > > This series allows to turn off and on the inflight-migration feature at runtime > > > without recreating the VM (it was the only way to turn off the feature, since > > > the protocol feature could no longer be turned off, after initialization with > > > the backend). This is more important feature, and it leads to the fact that > > > this feature allows to migrate between different versions of qemu (with and > > > without inflight-migration support). > > > > > > > > > --- > > > hw/block/vhost-user-blk.c | 10 ++++++++-- > > > 1 file changed, 8 insertions(+), 2 deletions(-) > > > > > > diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c > > > index e3b873af7c..650c004bde 100644 > > > --- a/hw/block/vhost-user-blk.c > > > +++ b/hw/block/vhost-user-blk.c > > > @@ -619,6 +619,8 @@ static const VMStateDescription vmstate_vhost_user_blk = { > > > } > > > }; > > > > > > +static PropertyInfo vhost_user_blk_inflight_migration_prop; > > > + > > > static const Property vhost_user_blk_properties[] = { > > > DEFINE_PROP_CHR("chardev", VHostUserBlk, chardev), > > > DEFINE_PROP_UINT16("num-queues", VHostUserBlk, num_queues, > > > @@ -632,8 +634,9 @@ static const Property vhost_user_blk_properties[] = { > > > VIRTIO_BLK_F_WRITE_ZEROES, true), > > > DEFINE_PROP_BOOL("skip-get-vring-base-on-force-shutdown", VHostUserBlk, > > > skip_get_vring_base_on_force_shutdown, false), > > > - DEFINE_PROP_BOOL("inflight-migration", VHostUserBlk, > > > - inflight_migration, false), > > > + DEFINE_PROP("inflight-migration", VHostUserBlk, inflight_migration, > > > + vhost_user_blk_inflight_migration_prop, bool, > > > + .set_default = true, .defval.u = false), > > > }; > > > > > > static void vhost_user_blk_class_init(ObjectClass *klass, const void *data) > > > @@ -665,6 +668,9 @@ static const TypeInfo vhost_user_blk_info = { > > > > > > static void virtio_register_types(void) > > > { > > > + vhost_user_blk_inflight_migration_prop = qdev_prop_bool; > > > + vhost_user_blk_inflight_migration_prop.realized_set_allowed = true; > > > + > > > type_register_static(&vhost_user_blk_info); > > > } > > > > > > So then, for example, let us say I paused the VM, then set the flag, > > > now inflight is on but GET_BASE did not drain it? > > > > > > If I understood the question correctly, this is valid behavior. Before > > > migration QEMU check protocol features to understand does the backend support > > > inflight migration. If it does, after that QEMU migrate inflight buffer to > > > other VM. If it's not, return error before migration started. > > > > I apologise i reverted the logic in the question. > > > > > > I start vm and inflight is on. > > I stop vm get base does not drain. > > In case of VM stop, backend wait to drain all requests. Ability to perform > drain or not available only during migration by skip_drain variable in > vhost_user_blk_stop(). What if it was migration but it failed? > skip_drain true only if inflight_migration is on and runstate is > FINISH_MIGRATION. This btw I don't much like, a stopped VM would preferably behave the same whatever the reason to stop. > > > i turn inflight off. > > now it looks like it will happily migrate? > > So in this case, when vm migrated with inflight off, it will leads to using > GET_VRING_BASE message, that wait all requests to be drained. > > > > > > -- > > > 2.34.1 > > > ^ permalink raw reply [flat|nested] 18+ messages in thread
[parent not found: <dd1f5830-75ff-4256-a179-6a8341a3b4ec@yandex-team.ru>]
* Re: [PATCH v5 4/6] vhost-user-blk: make inflight-migration prop mutable [not found] ` <dd1f5830-75ff-4256-a179-6a8341a3b4ec@yandex-team.ru> @ 2026-07-29 12:13 ` Michael S. Tsirkin 2026-07-29 14:07 ` Alexandr Moshkov 0 siblings, 1 reply; 18+ messages in thread From: Michael S. Tsirkin @ 2026-07-29 12:13 UTC (permalink / raw) To: Alexandr Moshkov Cc: qemu-devel, Hanna Reitz, Jason Wang, Vladimir Sementsov-Ogievskiy, Gonglei (Arei), Jason Wang, Fam Zheng, qemu-block, Stefan Hajnoczi, Alex Bennée, yc-core@yandex-team.ru, Pierrick Bouvier, Pierrick Bouvier, virtio-fs, Kevin Wolf, zhenwei pi, Paolo Bonzini, Stefano Garzarella, Milan Zamazal, Raphael Norwitz On Wed, Jul 29, 2026 at 03:56:54PM +0500, Alexandr Moshkov wrote: > > On 7/29/26 15:48, Michael S. Tsirkin wrote: > > On Wed, Jul 29, 2026 at 03:35:31PM +0500, Alexandr Moshkov wrote: > > On 7/29/26 15:14, Michael S. Tsirkin wrote: > > On Wed, Jul 29, 2026 at 03:09:57PM +0500, Alexandr Moshkov wrote: > > On 7/29/26 14:32, Michael S. Tsirkin wrote: > > On Tue, Jul 28, 2026 at 03:08:39PM +0500, Alexandr Moshkov wrote: > > When migrating from a QEMU version that supports inflight-migration to > an older one that does not, there is no way to disable the feature at > runtime — the VM must be stopped and reconfigured. This is impractical > in production environments. > > Make the inflight-migration property mutable after device realization > so it can be toggled via qom-set without restarting the VM. > > Acked-by: Raphael Norwitz <rnorwitz@nvidia.com> > Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru> > > > > I think I am beginning to understand. > > > You are running qemu with inflight-migration on and want to migrate > to qemu without inflight-migration at all. > > > Since it is guest transparent you could retrofit it like this. > > My question is why is it worth it, we do not normally support > migrating between qemu versions with different command lines. > > I think you right. Maybe I've been focusing too much on the ability to migrate > between versions of qemu with and without inflight migration. > > This series allows to turn off and on the inflight-migration feature at runtime > without recreating the VM restarting you mean. > (it was the only way to turn off the feature, since > the protocol feature could no longer be turned off, after initialization with > the backend). This is more important feature, and it leads to the fact that > this feature allows to migrate between different versions of qemu (with and > without inflight-migration support). so again it's not the 1st feature we have like this. we tie migration to the machine type and same set of command line flags specifically to keep things manageable. really cross version migration is a pain as it is. > > --- > hw/block/vhost-user-blk.c | 10 ++++++++-- > 1 file changed, 8 insertions(+), 2 deletions(-) > > diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c > index e3b873af7c..650c004bde 100644 > --- a/hw/block/vhost-user-blk.c > +++ b/hw/block/vhost-user-blk.c > @@ -619,6 +619,8 @@ static const VMStateDescription vmstate_vhost_user_blk = { > } > }; > > +static PropertyInfo vhost_user_blk_inflight_migration_prop; > + > static const Property vhost_user_blk_properties[] = { > DEFINE_PROP_CHR("chardev", VHostUserBlk, chardev), > DEFINE_PROP_UINT16("num-queues", VHostUserBlk, num_queues, > @@ -632,8 +634,9 @@ static const Property vhost_user_blk_properties[] = { > VIRTIO_BLK_F_WRITE_ZEROES, true), > DEFINE_PROP_BOOL("skip-get-vring-base-on-force-shutdown", VHostUserBlk, > skip_get_vring_base_on_force_shutdown, false), > - DEFINE_PROP_BOOL("inflight-migration", VHostUserBlk, > - inflight_migration, false), > + DEFINE_PROP("inflight-migration", VHostUserBlk, inflight_migration, > + vhost_user_blk_inflight_migration_prop, bool, > + .set_default = true, .defval.u = false), > }; > > static void vhost_user_blk_class_init(ObjectClass *klass, const void *data) > @@ -665,6 +668,9 @@ static const TypeInfo vhost_user_blk_info = { > > static void virtio_register_types(void) > { > + vhost_user_blk_inflight_migration_prop = qdev_prop_bool; > + vhost_user_blk_inflight_migration_prop.realized_set_allowed = true; > + > type_register_static(&vhost_user_blk_info); > } > > So then, for example, let us say I paused the VM, then set the flag, > now inflight is on but GET_BASE did not drain it? > > If I understood the question correctly, this is valid behavior. Before > migration QEMU check protocol features to understand does the backend support > inflight migration. If it does, after that QEMU migrate inflight buffer to > other VM. If it's not, return error before migration started. > > I apologise i reverted the logic in the question. > > > I start vm and inflight is on. > I stop vm get base does not drain. > > In case of VM stop, backend wait to drain all requests. Ability to perform > drain or not available only during migration by skip_drain variable in > vhost_user_blk_stop(). > > What if it was migration but it failed? > > If migration failed, source vm just keep using inflight region, backend > continue to execute inflight requests that was tried to migrate. Sorry, to be more clear. I mean migration succeeded but destination failed to start, so now vm is stopped and we are now trying migrating to a different destination. IIUC currently on VM stop we simply send GET_BASE and this stops backend, and depending on features things remain in the inflight buffer. Correct me if I am wrong. I think that we do not want a slow flush on vm stop if we can avoid it, and we also do not want backend to keep changing guest memory when VM is stopped. No? > > > skip_drain true only if inflight_migration is on and runstate is > FINISH_MIGRATION. FINISH_MIGRATE? actually i do not see where it affects it. > > This btw I don't much like, a stopped VM would preferably > behave the same whatever the reason to stop. > > Well, I don't see any other way to implement this. I am not 100% sure it's implementable as described, but I guess you could block changing the property if VM and thus the backend is not connected and running. Which is even more complexity but at least it is consistent. > I think backend must perform > drain in all cases except live migration with all the necessary checks > (protocol features, special message GET_VRING_BASE_SKIP_DRAIN). This is not what is going on now, right? Whether it drains depends on protocol features not on VM state and I think we should keep it like this. > > > i turn inflight off. > now it looks like it will happily migrate? > > So in this case, when vm migrated with inflight off, it will leads to using > GET_VRING_BASE message, that wait all requests to be drained. > > > -- > 2.34.1 > > i mean it migrated with on. vm was stopped then it flipped to off. -- MST ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v5 4/6] vhost-user-blk: make inflight-migration prop mutable 2026-07-29 12:13 ` Michael S. Tsirkin @ 2026-07-29 14:07 ` Alexandr Moshkov 2026-07-29 14:48 ` Michael S. Tsirkin 0 siblings, 1 reply; 18+ messages in thread From: Alexandr Moshkov @ 2026-07-29 14:07 UTC (permalink / raw) To: Michael S. Tsirkin Cc: qemu-devel, Hanna Reitz, Jason Wang, Vladimir Sementsov-Ogievskiy, Gonglei (Arei), Jason Wang, Fam Zheng, qemu-block, Stefan Hajnoczi, Alex Bennée, yc-core@yandex-team.ru, Pierrick Bouvier, Pierrick Bouvier, virtio-fs, Kevin Wolf, zhenwei pi, Paolo Bonzini, Stefano Garzarella, Milan Zamazal, Raphael Norwitz On 7/29/26 17:13, Michael S. Tsirkin wrote: > On Wed, Jul 29, 2026 at 03:56:54PM +0500, Alexandr Moshkov wrote: >> On 7/29/26 15:48, Michael S. Tsirkin wrote: >> >> On Wed, Jul 29, 2026 at 03:35:31PM +0500, Alexandr Moshkov wrote: >> >> On 7/29/26 15:14, Michael S. Tsirkin wrote: >> >> On Wed, Jul 29, 2026 at 03:09:57PM +0500, Alexandr Moshkov wrote: >> >> On 7/29/26 14:32, Michael S. Tsirkin wrote: >> >> On Tue, Jul 28, 2026 at 03:08:39PM +0500, Alexandr Moshkov wrote: >> >> When migrating from a QEMU version that supports inflight-migration to >> an older one that does not, there is no way to disable the feature at >> runtime — the VM must be stopped and reconfigured. This is impractical >> in production environments. >> >> Make the inflight-migration property mutable after device realization >> so it can be toggled via qom-set without restarting the VM. >> >> Acked-by: Raphael Norwitz <rnorwitz@nvidia.com> >> Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru> >> >> >> >> I think I am beginning to understand. >> >> >> You are running qemu with inflight-migration on and want to migrate >> to qemu without inflight-migration at all. >> >> >> Since it is guest transparent you could retrofit it like this. >> >> My question is why is it worth it, we do not normally support >> migrating between qemu versions with different command lines. >> >> I think you right. Maybe I've been focusing too much on the ability to migrate >> between versions of qemu with and without inflight migration. >> >> This series allows to turn off and on the inflight-migration feature at runtime >> without recreating the VM > restarting you mean. Yes, sorry. > >> (it was the only way to turn off the feature, since >> the protocol feature could no longer be turned off, after initialization with >> the backend). This is more important feature, and it leads to the fact that >> this feature allows to migrate between different versions of qemu (with and >> without inflight-migration support). > so again it's not the 1st feature we have like this. we tie migration > to the machine type and same set of command line flags > specifically to keep things manageable. > > really cross version migration is a pain as it is. I understand that, thanks! Maybe we should focus on the fact that this series allows to turn inflight_migration on and off (without cross version migration) ? Firstly, I wanted to implement this through migration capabilities, but in one of the threads, it was agreed that a separate capability only for vhost-user-blk is not a good idea. Therefore, I implemented it through a parameter that can be changed using qom-set. > >> --- >> hw/block/vhost-user-blk.c | 10 ++++++++-- >> 1 file changed, 8 insertions(+), 2 deletions(-) >> >> diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c >> index e3b873af7c..650c004bde 100644 >> --- a/hw/block/vhost-user-blk.c >> +++ b/hw/block/vhost-user-blk.c >> @@ -619,6 +619,8 @@ static const VMStateDescription vmstate_vhost_user_blk = { >> } >> }; >> >> +static PropertyInfo vhost_user_blk_inflight_migration_prop; >> + >> static const Property vhost_user_blk_properties[] = { >> DEFINE_PROP_CHR("chardev", VHostUserBlk, chardev), >> DEFINE_PROP_UINT16("num-queues", VHostUserBlk, num_queues, >> @@ -632,8 +634,9 @@ static const Property vhost_user_blk_properties[] = { >> VIRTIO_BLK_F_WRITE_ZEROES, true), >> DEFINE_PROP_BOOL("skip-get-vring-base-on-force-shutdown", VHostUserBlk, >> skip_get_vring_base_on_force_shutdown, false), >> - DEFINE_PROP_BOOL("inflight-migration", VHostUserBlk, >> - inflight_migration, false), >> + DEFINE_PROP("inflight-migration", VHostUserBlk, inflight_migration, >> + vhost_user_blk_inflight_migration_prop, bool, >> + .set_default = true, .defval.u = false), >> }; >> >> static void vhost_user_blk_class_init(ObjectClass *klass, const void *data) >> @@ -665,6 +668,9 @@ static const TypeInfo vhost_user_blk_info = { >> >> static void virtio_register_types(void) >> { >> + vhost_user_blk_inflight_migration_prop = qdev_prop_bool; >> + vhost_user_blk_inflight_migration_prop.realized_set_allowed = true; >> + >> type_register_static(&vhost_user_blk_info); >> } >> >> So then, for example, let us say I paused the VM, then set the flag, >> now inflight is on but GET_BASE did not drain it? >> >> If I understood the question correctly, this is valid behavior. Before >> migration QEMU check protocol features to understand does the backend support >> inflight migration. If it does, after that QEMU migrate inflight buffer to >> other VM. If it's not, return error before migration started. >> >> I apologise i reverted the logic in the question. >> >> >> I start vm and inflight is on. >> I stop vm get base does not drain. >> >> In case of VM stop, backend wait to drain all requests. Ability to perform >> drain or not available only during migration by skip_drain variable in >> vhost_user_blk_stop(). >> >> What if it was migration but it failed? >> >> If migration failed, source vm just keep using inflight region, backend >> continue to execute inflight requests that was tried to migrate. > > Sorry, to be more clear. I mean migration succeeded but destination failed to > start, so now vm is stopped and we are now trying migrating to a different destination. Is this a real case? I was thinking in case of any destination errors (migration process or starting), source vm has to work... I guess in that case, inflight requests can only be executed on different destination or in current destination if vm be able to start. > IIUC currently on VM stop we simply send GET_BASE and this stops > backend, and depending on features things remain in the inflight buffer. > Correct me if I am wrong. Yes backend has to disconnect from QEMU.. So it not be changing guest memory. > > I think that we do not want a slow flush on vm stop if we > can avoid it, and we also do not want backend to keep changing > guest memory when VM is stopped. No? Yes, it is >> >> skip_drain true only if inflight_migration is on and runstate is >> FINISH_MIGRATION. > FINISH_MIGRATE? actually i do not see where it affects it. Sorry, I didn't understand) Who affects who? > >> This btw I don't much like, a stopped VM would preferably >> behave the same whatever the reason to stop. >> >> Well, I don't see any other way to implement this. > > I am not 100% sure it's implementable as described, but I guess > you could block changing the property if VM and thus the backend > is not connected and running. > > > Which is even more complexity but at least it is consistent. Oh, I understand what you thinking about. I agreed, but why implementation (in this series) is not consistent? I was thinking that current (in current qemu version) impl is not, because GET_VRING_BASE has two behaviors, depending on the presence of the protocol feature (drain or not drain requests). Now, I think it would be more clear: one message for drain GET_VRING_BASE (as it was before), one for not drain GET_VRING_BASE_SKIP_DRAIN. > > > > > > >> I think backend must perform >> drain in all cases except live migration with all the necessary checks >> (protocol features, special message GET_VRING_BASE_SKIP_DRAIN). > > This is not what is going on now, right? Whether it drains depends > on protocol features not on VM state and I think we should > keep it like this. Well yes, now in current qemu version drain or not drain depends on protocol feature only, that try to be negotiated only if inflight_migration is set on. In patch: in case live migration (if protocol feature and device param) sent message GET_VRING_BASE_SKIP_DRAIN to skip drain, in other cases sent other message - GET_VRING_BASE with draining all requests. So drain or not drain depends on message that QEMU send to backend. > > >> >> i turn inflight off. >> now it looks like it will happily migrate? >> >> So in this case, when vm migrated with inflight off, it will leads to using >> GET_VRING_BASE message, that wait all requests to be drained. >> >> >> -- >> 2.34.1 >> >> > > i mean it migrated with on. vm was stopped then it flipped to off. After stop vm will send GET_VRING_BASE to backend, that lead to drain all requests. Non of the inflight requests will be missed here, If that's what you mean. > > > > > ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v5 4/6] vhost-user-blk: make inflight-migration prop mutable 2026-07-29 14:07 ` Alexandr Moshkov @ 2026-07-29 14:48 ` Michael S. Tsirkin 2026-07-29 15:18 ` Alexandr Moshkov 0 siblings, 1 reply; 18+ messages in thread From: Michael S. Tsirkin @ 2026-07-29 14:48 UTC (permalink / raw) To: Alexandr Moshkov Cc: qemu-devel, Hanna Reitz, Jason Wang, Vladimir Sementsov-Ogievskiy, Gonglei (Arei), Jason Wang, Fam Zheng, qemu-block, Stefan Hajnoczi, Alex Bennée, yc-core@yandex-team.ru, Pierrick Bouvier, Pierrick Bouvier, virtio-fs, Kevin Wolf, zhenwei pi, Paolo Bonzini, Stefano Garzarella, Milan Zamazal, Raphael Norwitz On Wed, Jul 29, 2026 at 07:07:11PM +0500, Alexandr Moshkov wrote: > > On 7/29/26 17:13, Michael S. Tsirkin wrote: > > On Wed, Jul 29, 2026 at 03:56:54PM +0500, Alexandr Moshkov wrote: > > > On 7/29/26 15:48, Michael S. Tsirkin wrote: > > > > > > On Wed, Jul 29, 2026 at 03:35:31PM +0500, Alexandr Moshkov wrote: > > > > > > On 7/29/26 15:14, Michael S. Tsirkin wrote: > > > > > > On Wed, Jul 29, 2026 at 03:09:57PM +0500, Alexandr Moshkov wrote: > > > > > > On 7/29/26 14:32, Michael S. Tsirkin wrote: > > > > > > On Tue, Jul 28, 2026 at 03:08:39PM +0500, Alexandr Moshkov wrote: > > > > > > When migrating from a QEMU version that supports inflight-migration to > > > an older one that does not, there is no way to disable the feature at > > > runtime — the VM must be stopped and reconfigured. This is impractical > > > in production environments. > > > > > > Make the inflight-migration property mutable after device realization > > > so it can be toggled via qom-set without restarting the VM. > > > > > > Acked-by: Raphael Norwitz <rnorwitz@nvidia.com> > > > Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru> > > > > > > > > > > > > I think I am beginning to understand. > > > > > > > > > You are running qemu with inflight-migration on and want to migrate > > > to qemu without inflight-migration at all. > > > > > > > > > Since it is guest transparent you could retrofit it like this. > > > > > > My question is why is it worth it, we do not normally support > > > migrating between qemu versions with different command lines. > > > > > > I think you right. Maybe I've been focusing too much on the ability to migrate > > > between versions of qemu with and without inflight migration. > > > > > > This series allows to turn off and on the inflight-migration feature at runtime > > > without recreating the VM > > restarting you mean. > Yes, sorry. > > > > > (it was the only way to turn off the feature, since > > > the protocol feature could no longer be turned off, after initialization with > > > the backend). This is more important feature, and it leads to the fact that > > > this feature allows to migrate between different versions of qemu (with and > > > without inflight-migration support). > > so again it's not the 1st feature we have like this. we tie migration > > to the machine type and same set of command line flags > > specifically to keep things manageable. > > > > really cross version migration is a pain as it is. > > I understand that, thanks! Maybe we should focus on the fact that this > series allows to turn inflight_migration on and off (without cross version > migration) ? > > Firstly, I wanted to implement this through migration capabilities, but in > one of the threads, > it was agreed that a separate capability only for vhost-user-blk is not a > good idea. I mean it's a generic thing. > Therefore, I implemented it through a parameter that can be changed using > qom-set. > > > > > > --- > > > hw/block/vhost-user-blk.c | 10 ++++++++-- > > > 1 file changed, 8 insertions(+), 2 deletions(-) > > > > > > diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c > > > index e3b873af7c..650c004bde 100644 > > > --- a/hw/block/vhost-user-blk.c > > > +++ b/hw/block/vhost-user-blk.c > > > @@ -619,6 +619,8 @@ static const VMStateDescription vmstate_vhost_user_blk = { > > > } > > > }; > > > > > > +static PropertyInfo vhost_user_blk_inflight_migration_prop; > > > + > > > static const Property vhost_user_blk_properties[] = { > > > DEFINE_PROP_CHR("chardev", VHostUserBlk, chardev), > > > DEFINE_PROP_UINT16("num-queues", VHostUserBlk, num_queues, > > > @@ -632,8 +634,9 @@ static const Property vhost_user_blk_properties[] = { > > > VIRTIO_BLK_F_WRITE_ZEROES, true), > > > DEFINE_PROP_BOOL("skip-get-vring-base-on-force-shutdown", VHostUserBlk, > > > skip_get_vring_base_on_force_shutdown, false), > > > - DEFINE_PROP_BOOL("inflight-migration", VHostUserBlk, > > > - inflight_migration, false), > > > + DEFINE_PROP("inflight-migration", VHostUserBlk, inflight_migration, > > > + vhost_user_blk_inflight_migration_prop, bool, > > > + .set_default = true, .defval.u = false), > > > }; > > > > > > static void vhost_user_blk_class_init(ObjectClass *klass, const void *data) > > > @@ -665,6 +668,9 @@ static const TypeInfo vhost_user_blk_info = { > > > > > > static void virtio_register_types(void) > > > { > > > + vhost_user_blk_inflight_migration_prop = qdev_prop_bool; > > > + vhost_user_blk_inflight_migration_prop.realized_set_allowed = true; > > > + > > > type_register_static(&vhost_user_blk_info); > > > } > > > > > > So then, for example, let us say I paused the VM, then set the flag, > > > now inflight is on but GET_BASE did not drain it? > > > > > > If I understood the question correctly, this is valid behavior. Before > > > migration QEMU check protocol features to understand does the backend support > > > inflight migration. If it does, after that QEMU migrate inflight buffer to > > > other VM. If it's not, return error before migration started. > > > > > > I apologise i reverted the logic in the question. > > > > > > > > > I start vm and inflight is on. > > > I stop vm get base does not drain. > > > > > > In case of VM stop, backend wait to drain all requests. Ability to perform > > > drain or not available only during migration by skip_drain variable in > > > vhost_user_blk_stop(). > > > > > > What if it was migration but it failed? > > > > > > If migration failed, source vm just keep using inflight region, backend > > > continue to execute inflight requests that was tried to migrate. > > > > Sorry, to be more clear. I mean migration succeeded but destination failed to > > start, so now vm is stopped and we are now trying migrating to a different destination. > > Is this a real case? I was thinking in case of any destination errors > (migration process or starting), source vm has to work... It does not start automatically in all cases, no. > I guess in that case, inflight requests can only be executed on different > destination or in current destination if vm be able to start. > > > IIUC currently on VM stop we simply send GET_BASE and this stops > > backend, and depending on features things remain in the inflight buffer. > > Correct me if I am wrong. > Yes backend has to disconnect from QEMU.. So it not be changing guest > memory. > > > > I think that we do not want a slow flush on vm stop if we > > can avoid it, and we also do not want backend to keep changing > > guest memory when VM is stopped. No? > Yes, it is > > > > > > skip_drain true only if inflight_migration is on and runstate is > > > FINISH_MIGRATION. > > FINISH_MIGRATE? actually i do not see where it affects it. > > Sorry, I didn't understand) Who affects who? > > > > > This btw I don't much like, a stopped VM would preferably > > > behave the same whatever the reason to stop. > > > > > > Well, I don't see any other way to implement this. > > > > I am not 100% sure it's implementable as described, but I guess > > you could block changing the property if VM and thus the backend > > is not connected and running. > > > > > > Which is even more complexity but at least it is consistent. > > Oh, I understand what you thinking about. > > I agreed, but why implementation (in this series) is not consistent? I was > thinking that current (in current qemu version) impl is not, because > GET_VRING_BASE has two behaviors, depending on the presence of the protocol > feature (drain or not drain requests). > > Now, I think it would be more clear: one message for drain GET_VRING_BASE > (as it was before), one for not drain GET_VRING_BASE_SKIP_DRAIN. Well the message is consistent. But its use does not seem to be. > > > > > > > > > > > > > > > I think backend must perform > > > drain in all cases except live migration with all the necessary checks > > > (protocol features, special message GET_VRING_BASE_SKIP_DRAIN). > > > > This is not what is going on now, right? Whether it drains depends > > on protocol features not on VM state and I think we should > > keep it like this. > > Well yes, now in current qemu version drain or not drain depends on protocol > feature only, that try to be negotiated only if inflight_migration is set > on. > > In patch: in case live migration (if protocol feature and device param) > sent message GET_VRING_BASE_SKIP_DRAIN to skip drain, in other cases sent > other message - GET_VRING_BASE with draining all requests. > So drain or not drain depends on message that QEMU send to backend. > > > > > > > > > > > i turn inflight off. > > > now it looks like it will happily migrate? > > > > > > So in this case, when vm migrated with inflight off, it will leads to using > > > GET_VRING_BASE message, that wait all requests to be drained. > > > > > > > > > -- > > > 2.34.1 > > > > > > > > > > i mean it migrated with on. vm was stopped then it flipped to off. > After stop vm will send GET_VRING_BASE to backend, that lead to drain all > requests. Non of the inflight requests will be missed here, If that's what > you mean. this: + bool skip_drain = vhost_user_blk_inflight_needed(s) && + runstate_check(RUN_STATE_FINISH_MIGRATE); is what I dislike. I dislike poking at migration state. It's painful enough that we need to worry about VM running or not. > > > > > > > > > > ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v5 4/6] vhost-user-blk: make inflight-migration prop mutable 2026-07-29 14:48 ` Michael S. Tsirkin @ 2026-07-29 15:18 ` Alexandr Moshkov 2026-07-29 18:50 ` Michael S. Tsirkin 0 siblings, 1 reply; 18+ messages in thread From: Alexandr Moshkov @ 2026-07-29 15:18 UTC (permalink / raw) To: Michael S. Tsirkin Cc: qemu-devel, Hanna Reitz, Jason Wang, Vladimir Sementsov-Ogievskiy, Gonglei (Arei), Jason Wang, Fam Zheng, qemu-block, Stefan Hajnoczi, Alex Bennée, yc-core@yandex-team.ru, Pierrick Bouvier, Pierrick Bouvier, virtio-fs, Kevin Wolf, zhenwei pi, Paolo Bonzini, Stefano Garzarella, Milan Zamazal, Raphael Norwitz On 7/29/26 19:48, Michael S. Tsirkin wrote: > On Wed, Jul 29, 2026 at 07:07:11PM +0500, Alexandr Moshkov wrote: >> On 7/29/26 17:13, Michael S. Tsirkin wrote: >>> On Wed, Jul 29, 2026 at 03:56:54PM +0500, Alexandr Moshkov wrote: >>>> On 7/29/26 15:48, Michael S. Tsirkin wrote: >>>> >>>> On Wed, Jul 29, 2026 at 03:35:31PM +0500, Alexandr Moshkov wrote: >>>> >>>> On 7/29/26 15:14, Michael S. Tsirkin wrote: >>>> >>>> On Wed, Jul 29, 2026 at 03:09:57PM +0500, Alexandr Moshkov wrote: >>>> >>>> On 7/29/26 14:32, Michael S. Tsirkin wrote: >>>> >>>> On Tue, Jul 28, 2026 at 03:08:39PM +0500, Alexandr Moshkov wrote: >>>> >>>> When migrating from a QEMU version that supports inflight-migration to >>>> an older one that does not, there is no way to disable the feature at >>>> runtime — the VM must be stopped and reconfigured. This is impractical >>>> in production environments. >>>> >>>> Make the inflight-migration property mutable after device realization >>>> so it can be toggled via qom-set without restarting the VM. >>>> >>>> Acked-by: Raphael Norwitz <rnorwitz@nvidia.com> >>>> Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru> >>>> >>>> >>>> >>>> I think I am beginning to understand. >>>> >>>> >>>> You are running qemu with inflight-migration on and want to migrate >>>> to qemu without inflight-migration at all. >>>> >>>> >>>> Since it is guest transparent you could retrofit it like this. >>>> >>>> My question is why is it worth it, we do not normally support >>>> migrating between qemu versions with different command lines. >>>> >>>> I think you right. Maybe I've been focusing too much on the ability to migrate >>>> between versions of qemu with and without inflight migration. >>>> >>>> This series allows to turn off and on the inflight-migration feature at runtime >>>> without recreating the VM >>> restarting you mean. >> Yes, sorry. >>>> (it was the only way to turn off the feature, since >>>> the protocol feature could no longer be turned off, after initialization with >>>> the backend). This is more important feature, and it leads to the fact that >>>> this feature allows to migrate between different versions of qemu (with and >>>> without inflight-migration support). >>> so again it's not the 1st feature we have like this. we tie migration >>> to the machine type and same set of command line flags >>> specifically to keep things manageable. >>> >>> really cross version migration is a pain as it is. >> I understand that, thanks! Maybe we should focus on the fact that this >> series allows to turn inflight_migration on and off (without cross version >> migration) ? >> >> Firstly, I wanted to implement this through migration capabilities, but in >> one of the threads, >> it was agreed that a separate capability only for vhost-user-blk is not a >> good idea. > I mean it's a generic thing. > >> Therefore, I implemented it through a parameter that can be changed using >> qom-set. >> >>>> --- >>>> hw/block/vhost-user-blk.c | 10 ++++++++-- >>>> 1 file changed, 8 insertions(+), 2 deletions(-) >>>> >>>> diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c >>>> index e3b873af7c..650c004bde 100644 >>>> --- a/hw/block/vhost-user-blk.c >>>> +++ b/hw/block/vhost-user-blk.c >>>> @@ -619,6 +619,8 @@ static const VMStateDescription vmstate_vhost_user_blk = { >>>> } >>>> }; >>>> >>>> +static PropertyInfo vhost_user_blk_inflight_migration_prop; >>>> + >>>> static const Property vhost_user_blk_properties[] = { >>>> DEFINE_PROP_CHR("chardev", VHostUserBlk, chardev), >>>> DEFINE_PROP_UINT16("num-queues", VHostUserBlk, num_queues, >>>> @@ -632,8 +634,9 @@ static const Property vhost_user_blk_properties[] = { >>>> VIRTIO_BLK_F_WRITE_ZEROES, true), >>>> DEFINE_PROP_BOOL("skip-get-vring-base-on-force-shutdown", VHostUserBlk, >>>> skip_get_vring_base_on_force_shutdown, false), >>>> - DEFINE_PROP_BOOL("inflight-migration", VHostUserBlk, >>>> - inflight_migration, false), >>>> + DEFINE_PROP("inflight-migration", VHostUserBlk, inflight_migration, >>>> + vhost_user_blk_inflight_migration_prop, bool, >>>> + .set_default = true, .defval.u = false), >>>> }; >>>> >>>> static void vhost_user_blk_class_init(ObjectClass *klass, const void *data) >>>> @@ -665,6 +668,9 @@ static const TypeInfo vhost_user_blk_info = { >>>> >>>> static void virtio_register_types(void) >>>> { >>>> + vhost_user_blk_inflight_migration_prop = qdev_prop_bool; >>>> + vhost_user_blk_inflight_migration_prop.realized_set_allowed = true; >>>> + >>>> type_register_static(&vhost_user_blk_info); >>>> } >>>> >>>> So then, for example, let us say I paused the VM, then set the flag, >>>> now inflight is on but GET_BASE did not drain it? >>>> >>>> If I understood the question correctly, this is valid behavior. Before >>>> migration QEMU check protocol features to understand does the backend support >>>> inflight migration. If it does, after that QEMU migrate inflight buffer to >>>> other VM. If it's not, return error before migration started. >>>> >>>> I apologise i reverted the logic in the question. >>>> >>>> >>>> I start vm and inflight is on. >>>> I stop vm get base does not drain. >>>> >>>> In case of VM stop, backend wait to drain all requests. Ability to perform >>>> drain or not available only during migration by skip_drain variable in >>>> vhost_user_blk_stop(). >>>> >>>> What if it was migration but it failed? >>>> >>>> If migration failed, source vm just keep using inflight region, backend >>>> continue to execute inflight requests that was tried to migrate. >>> Sorry, to be more clear. I mean migration succeeded but destination failed to >>> start, so now vm is stopped and we are now trying migrating to a different destination. >> Is this a real case? I was thinking in case of any destination errors >> (migration process or starting), source vm has to work... > It does not start automatically in all cases, no. > >> I guess in that case, inflight requests can only be executed on different >> destination or in current destination if vm be able to start. >> >>> IIUC currently on VM stop we simply send GET_BASE and this stops >>> backend, and depending on features things remain in the inflight buffer. >>> Correct me if I am wrong. >> Yes backend has to disconnect from QEMU.. So it not be changing guest >> memory. >>> I think that we do not want a slow flush on vm stop if we >>> can avoid it, and we also do not want backend to keep changing >>> guest memory when VM is stopped. No? >> Yes, it is >>>> skip_drain true only if inflight_migration is on and runstate is >>>> FINISH_MIGRATION. >>> FINISH_MIGRATE? actually i do not see where it affects it. >> Sorry, I didn't understand) Who affects who? > >>>> This btw I don't much like, a stopped VM would preferably >>>> behave the same whatever the reason to stop. >>>> >>>> Well, I don't see any other way to implement this. >>> I am not 100% sure it's implementable as described, but I guess >>> you could block changing the property if VM and thus the backend >>> is not connected and running. >>> >>> >>> Which is even more complexity but at least it is consistent. >> Oh, I understand what you thinking about. >> >> I agreed, but why implementation (in this series) is not consistent? I was >> thinking that current (in current qemu version) impl is not, because >> GET_VRING_BASE has two behaviors, depending on the presence of the protocol >> feature (drain or not drain requests). >> >> Now, I think it would be more clear: one message for drain GET_VRING_BASE >> (as it was before), one for not drain GET_VRING_BASE_SKIP_DRAIN. > > Well the message is consistent. But its use does not seem to be. > >>> >>> >>> >>> >>> >>>> I think backend must perform >>>> drain in all cases except live migration with all the necessary checks >>>> (protocol features, special message GET_VRING_BASE_SKIP_DRAIN). >>> This is not what is going on now, right? Whether it drains depends >>> on protocol features not on VM state and I think we should >>> keep it like this. >> Well yes, now in current qemu version drain or not drain depends on protocol >> feature only, that try to be negotiated only if inflight_migration is set >> on. >> >> In patch: in case live migration (if protocol feature and device param) >> sent message GET_VRING_BASE_SKIP_DRAIN to skip drain, in other cases sent >> other message - GET_VRING_BASE with draining all requests. >> So drain or not drain depends on message that QEMU send to backend. >> >>> >>>> i turn inflight off. >>>> now it looks like it will happily migrate? >>>> >>>> So in this case, when vm migrated with inflight off, it will leads to using >>>> GET_VRING_BASE message, that wait all requests to be drained. >>>> >>>> >>>> -- >>>> 2.34.1 >>>> >>>> >>> i mean it migrated with on. vm was stopped then it flipped to off. >> After stop vm will send GET_VRING_BASE to backend, that lead to drain all >> requests. Non of the inflight requests will be missed here, If that's what >> you mean. > this: > > + bool skip_drain = vhost_user_blk_inflight_needed(s) && > + runstate_check(RUN_STATE_FINISH_MIGRATE); > > is what I dislike. > > I dislike poking at migration state. It's painful enough that we > need to worry about VM running or not. I understand that. But I don't know how to do it another way. We can't just erase runstate check - it will possibly leads to missing inflight requests during VM stop. And don't know how to make stop more consistent. Do you have any thoughts? >>> >>> >>> >>> ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v5 4/6] vhost-user-blk: make inflight-migration prop mutable 2026-07-29 15:18 ` Alexandr Moshkov @ 2026-07-29 18:50 ` Michael S. Tsirkin 0 siblings, 0 replies; 18+ messages in thread From: Michael S. Tsirkin @ 2026-07-29 18:50 UTC (permalink / raw) To: Alexandr Moshkov Cc: qemu-devel, Hanna Reitz, Jason Wang, Vladimir Sementsov-Ogievskiy, Gonglei (Arei), Jason Wang, Fam Zheng, qemu-block, Stefan Hajnoczi, Alex Bennée, yc-core@yandex-team.ru, Pierrick Bouvier, Pierrick Bouvier, virtio-fs, Kevin Wolf, zhenwei pi, Paolo Bonzini, Stefano Garzarella, Milan Zamazal, Raphael Norwitz On Wed, Jul 29, 2026 at 08:18:19PM +0500, Alexandr Moshkov wrote: > > On 7/29/26 19:48, Michael S. Tsirkin wrote: > > On Wed, Jul 29, 2026 at 07:07:11PM +0500, Alexandr Moshkov wrote: > > > On 7/29/26 17:13, Michael S. Tsirkin wrote: > > > > On Wed, Jul 29, 2026 at 03:56:54PM +0500, Alexandr Moshkov wrote: > > > > > On 7/29/26 15:48, Michael S. Tsirkin wrote: > > > > > > > > > > On Wed, Jul 29, 2026 at 03:35:31PM +0500, Alexandr Moshkov wrote: > > > > > > > > > > On 7/29/26 15:14, Michael S. Tsirkin wrote: > > > > > > > > > > On Wed, Jul 29, 2026 at 03:09:57PM +0500, Alexandr Moshkov wrote: > > > > > > > > > > On 7/29/26 14:32, Michael S. Tsirkin wrote: > > > > > > > > > > On Tue, Jul 28, 2026 at 03:08:39PM +0500, Alexandr Moshkov wrote: > > > > > > > > > > When migrating from a QEMU version that supports inflight-migration to > > > > > an older one that does not, there is no way to disable the feature at > > > > > runtime — the VM must be stopped and reconfigured. This is impractical > > > > > in production environments. > > > > > > > > > > Make the inflight-migration property mutable after device realization > > > > > so it can be toggled via qom-set without restarting the VM. > > > > > > > > > > Acked-by: Raphael Norwitz <rnorwitz@nvidia.com> > > > > > Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru> > > > > > > > > > > > > > > > > > > > > I think I am beginning to understand. > > > > > > > > > > > > > > > You are running qemu with inflight-migration on and want to migrate > > > > > to qemu without inflight-migration at all. > > > > > > > > > > > > > > > Since it is guest transparent you could retrofit it like this. > > > > > > > > > > My question is why is it worth it, we do not normally support > > > > > migrating between qemu versions with different command lines. > > > > > > > > > > I think you right. Maybe I've been focusing too much on the ability to migrate > > > > > between versions of qemu with and without inflight migration. > > > > > > > > > > This series allows to turn off and on the inflight-migration feature at runtime > > > > > without recreating the VM > > > > restarting you mean. > > > Yes, sorry. > > > > > (it was the only way to turn off the feature, since > > > > > the protocol feature could no longer be turned off, after initialization with > > > > > the backend). This is more important feature, and it leads to the fact that > > > > > this feature allows to migrate between different versions of qemu (with and > > > > > without inflight-migration support). > > > > so again it's not the 1st feature we have like this. we tie migration > > > > to the machine type and same set of command line flags > > > > specifically to keep things manageable. > > > > > > > > really cross version migration is a pain as it is. > > > I understand that, thanks! Maybe we should focus on the fact that this > > > series allows to turn inflight_migration on and off (without cross version > > > migration) ? > > > > > > Firstly, I wanted to implement this through migration capabilities, but in > > > one of the threads, > > > it was agreed that a separate capability only for vhost-user-blk is not a > > > good idea. > > I mean it's a generic thing. > > > > > Therefore, I implemented it through a parameter that can be changed using > > > qom-set. > > > > > > > > --- > > > > > hw/block/vhost-user-blk.c | 10 ++++++++-- > > > > > 1 file changed, 8 insertions(+), 2 deletions(-) > > > > > > > > > > diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c > > > > > index e3b873af7c..650c004bde 100644 > > > > > --- a/hw/block/vhost-user-blk.c > > > > > +++ b/hw/block/vhost-user-blk.c > > > > > @@ -619,6 +619,8 @@ static const VMStateDescription vmstate_vhost_user_blk = { > > > > > } > > > > > }; > > > > > > > > > > +static PropertyInfo vhost_user_blk_inflight_migration_prop; > > > > > + > > > > > static const Property vhost_user_blk_properties[] = { > > > > > DEFINE_PROP_CHR("chardev", VHostUserBlk, chardev), > > > > > DEFINE_PROP_UINT16("num-queues", VHostUserBlk, num_queues, > > > > > @@ -632,8 +634,9 @@ static const Property vhost_user_blk_properties[] = { > > > > > VIRTIO_BLK_F_WRITE_ZEROES, true), > > > > > DEFINE_PROP_BOOL("skip-get-vring-base-on-force-shutdown", VHostUserBlk, > > > > > skip_get_vring_base_on_force_shutdown, false), > > > > > - DEFINE_PROP_BOOL("inflight-migration", VHostUserBlk, > > > > > - inflight_migration, false), > > > > > + DEFINE_PROP("inflight-migration", VHostUserBlk, inflight_migration, > > > > > + vhost_user_blk_inflight_migration_prop, bool, > > > > > + .set_default = true, .defval.u = false), > > > > > }; > > > > > > > > > > static void vhost_user_blk_class_init(ObjectClass *klass, const void *data) > > > > > @@ -665,6 +668,9 @@ static const TypeInfo vhost_user_blk_info = { > > > > > > > > > > static void virtio_register_types(void) > > > > > { > > > > > + vhost_user_blk_inflight_migration_prop = qdev_prop_bool; > > > > > + vhost_user_blk_inflight_migration_prop.realized_set_allowed = true; > > > > > + > > > > > type_register_static(&vhost_user_blk_info); > > > > > } > > > > > > > > > > So then, for example, let us say I paused the VM, then set the flag, > > > > > now inflight is on but GET_BASE did not drain it? > > > > > > > > > > If I understood the question correctly, this is valid behavior. Before > > > > > migration QEMU check protocol features to understand does the backend support > > > > > inflight migration. If it does, after that QEMU migrate inflight buffer to > > > > > other VM. If it's not, return error before migration started. > > > > > > > > > > I apologise i reverted the logic in the question. > > > > > > > > > > > > > > > I start vm and inflight is on. > > > > > I stop vm get base does not drain. > > > > > > > > > > In case of VM stop, backend wait to drain all requests. Ability to perform > > > > > drain or not available only during migration by skip_drain variable in > > > > > vhost_user_blk_stop(). > > > > > > > > > > What if it was migration but it failed? > > > > > > > > > > If migration failed, source vm just keep using inflight region, backend > > > > > continue to execute inflight requests that was tried to migrate. > > > > Sorry, to be more clear. I mean migration succeeded but destination failed to > > > > start, so now vm is stopped and we are now trying migrating to a different destination. > > > Is this a real case? I was thinking in case of any destination errors > > > (migration process or starting), source vm has to work... > > It does not start automatically in all cases, no. > > > > > I guess in that case, inflight requests can only be executed on different > > > destination or in current destination if vm be able to start. > > > > > > > IIUC currently on VM stop we simply send GET_BASE and this stops > > > > backend, and depending on features things remain in the inflight buffer. > > > > Correct me if I am wrong. > > > Yes backend has to disconnect from QEMU.. So it not be changing guest > > > memory. > > > > I think that we do not want a slow flush on vm stop if we > > > > can avoid it, and we also do not want backend to keep changing > > > > guest memory when VM is stopped. No? > > > Yes, it is > > > > > skip_drain true only if inflight_migration is on and runstate is > > > > > FINISH_MIGRATION. > > > > FINISH_MIGRATE? actually i do not see where it affects it. > > > Sorry, I didn't understand) Who affects who? > > > > > > > This btw I don't much like, a stopped VM would preferably > > > > > behave the same whatever the reason to stop. > > > > > > > > > > Well, I don't see any other way to implement this. > > > > I am not 100% sure it's implementable as described, but I guess > > > > you could block changing the property if VM and thus the backend > > > > is not connected and running. > > > > > > > > > > > > Which is even more complexity but at least it is consistent. > > > Oh, I understand what you thinking about. > > > > > > I agreed, but why implementation (in this series) is not consistent? I was > > > thinking that current (in current qemu version) impl is not, because > > > GET_VRING_BASE has two behaviors, depending on the presence of the protocol > > > feature (drain or not drain requests). > > > > > > Now, I think it would be more clear: one message for drain GET_VRING_BASE > > > (as it was before), one for not drain GET_VRING_BASE_SKIP_DRAIN. > > > > Well the message is consistent. But its use does not seem to be. > > > > > > > > > > > > > > > > > > > > > > > > > > > I think backend must perform > > > > > drain in all cases except live migration with all the necessary checks > > > > > (protocol features, special message GET_VRING_BASE_SKIP_DRAIN). > > > > This is not what is going on now, right? Whether it drains depends > > > > on protocol features not on VM state and I think we should > > > > keep it like this. > > > Well yes, now in current qemu version drain or not drain depends on protocol > > > feature only, that try to be negotiated only if inflight_migration is set > > > on. > > > > > > In patch: in case live migration (if protocol feature and device param) > > > sent message GET_VRING_BASE_SKIP_DRAIN to skip drain, in other cases sent > > > other message - GET_VRING_BASE with draining all requests. > > > So drain or not drain depends on message that QEMU send to backend. > > > > > > > > > > > > i turn inflight off. > > > > > now it looks like it will happily migrate? > > > > > > > > > > So in this case, when vm migrated with inflight off, it will leads to using > > > > > GET_VRING_BASE message, that wait all requests to be drained. > > > > > > > > > > > > > > > -- > > > > > 2.34.1 > > > > > > > > > > > > > > i mean it migrated with on. vm was stopped then it flipped to off. > > > After stop vm will send GET_VRING_BASE to backend, that lead to drain all > > > requests. Non of the inflight requests will be missed here, If that's what > > > you mean. > > this: > > > > + bool skip_drain = vhost_user_blk_inflight_needed(s) && > > + runstate_check(RUN_STATE_FINISH_MIGRATE); > > > > is what I dislike. > > > > I dislike poking at migration state. It's painful enough that we > > need to worry about VM running or not. > > I understand that. But I don't know how to do it another way. We can't just > erase runstate check - it will possibly leads to missing inflight requests > during VM stop. > > And don't know how to make stop more consistent. Do you have any thoughts? Block disabling this when VM is not running. > > > > > > > > > > > > > > > > ^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH v5 5/6] vhost-user-blk: move inflight_needed higher 2026-07-28 10:08 [PATCH v5 0/6] vhost-user-blk: add compatibility with older qemu versions Alexandr Moshkov ` (3 preceding siblings ...) 2026-07-28 10:08 ` [PATCH v5 4/6] vhost-user-blk: make inflight-migration prop mutable Alexandr Moshkov @ 2026-07-28 10:08 ` Alexandr Moshkov 2026-07-28 10:08 ` [PATCH v5 6/6] vhost-user-blk: use GET_VRING_BASE_SKIP_DRAIN during inflight migration Alexandr Moshkov 2026-07-29 9:14 ` [PATCH v5 0/6] vhost-user-blk: add compatibility with older qemu versions Michael S. Tsirkin 6 siblings, 0 replies; 18+ messages in thread From: Alexandr Moshkov @ 2026-07-28 10:08 UTC (permalink / raw) To: qemu-devel Cc: Hanna Reitz, Jason Wang, Vladimir Sementsov-Ogievskiy, Gonglei (Arei), Jason Wang, Fam Zheng, qemu-block, Stefan Hajnoczi, Alex Bennée, yc-core@yandex-team.ru, Pierrick Bouvier, Pierrick Bouvier, Michael S. Tsirkin, virtio-fs, Kevin Wolf, zhenwei pi, Paolo Bonzini, Stefano Garzarella, Milan Zamazal, Raphael Norwitz, Alexandr Moshkov Move vhost_user_blk_inflight_needed() earlier in the file so it can be called from vhost_user_blk_stop(), which is defined before the VMState section. No functional change. Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru> Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru> --- hw/block/vhost-user-blk.c | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c index 650c004bde..156ed6de30 100644 --- a/hw/block/vhost-user-blk.c +++ b/hw/block/vhost-user-blk.c @@ -131,6 +131,17 @@ const VhostDevConfigOps blk_ops = { .vhost_dev_config_notifier = vhost_user_blk_handle_config_change, }; +static bool vhost_user_blk_inflight_needed(void *opaque) +{ + struct VHostUserBlk *s = opaque; + + bool inflight_migration = virtio_has_feature(s->dev.protocol_features, + VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT); + + return inflight_migration; +} + + static int vhost_user_blk_start(VirtIODevice *vdev, Error **errp) { VHostUserBlk *s = VHOST_USER_BLK(vdev); @@ -587,14 +598,6 @@ static struct vhost_dev *vhost_user_blk_get_vhost(VirtIODevice *vdev) return &s->dev; } -static bool vhost_user_blk_inflight_needed(void *opaque) -{ - struct VHostUserBlk *s = opaque; - - return vhost_user_has_protocol_feature( - &s->dev, VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT); -} - static const VMStateDescription vmstate_vhost_user_blk_inflight = { .name = "vhost-user-blk/inflight", .version_id = 1, -- 2.34.1 ^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH v5 6/6] vhost-user-blk: use GET_VRING_BASE_SKIP_DRAIN during inflight migration 2026-07-28 10:08 [PATCH v5 0/6] vhost-user-blk: add compatibility with older qemu versions Alexandr Moshkov ` (4 preceding siblings ...) 2026-07-28 10:08 ` [PATCH v5 5/6] vhost-user-blk: move inflight_needed higher Alexandr Moshkov @ 2026-07-28 10:08 ` Alexandr Moshkov 2026-07-29 9:14 ` [PATCH v5 0/6] vhost-user-blk: add compatibility with older qemu versions Michael S. Tsirkin 6 siblings, 0 replies; 18+ messages in thread From: Alexandr Moshkov @ 2026-07-28 10:08 UTC (permalink / raw) To: qemu-devel Cc: Hanna Reitz, Jason Wang, Vladimir Sementsov-Ogievskiy, Gonglei (Arei), Jason Wang, Fam Zheng, qemu-block, Stefan Hajnoczi, Alex Bennée, yc-core@yandex-team.ru, Pierrick Bouvier, Pierrick Bouvier, Michael S. Tsirkin, virtio-fs, Kevin Wolf, zhenwei pi, Paolo Bonzini, Stefano Garzarella, Milan Zamazal, Raphael Norwitz, Alexandr Moshkov Currently during live migration vhost-user-blk sends GET_VRING_BASE to stop vrings, which causes the back-end to drain all in-flight I/O before returning. This blocks the migration source until all I/O completes, adding significant downtime proportional to the I/O load. When inflight-migration is enabled and the device is stopping due to live migration (RUN_STATE_FINISH_MIGRATE), send GET_VRING_BASE_SKIP_DRAIN instead. This instructs the back-end to immediately suspend in-flight I/O and record it in the shared inflight region, which is then migrated to the destination host along with the rest of the device state. If the back-end does not support VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN, migration is aborted with an error in pre_save rather than hitting an assert at runtime. Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru> --- hw/block/vhost-user-blk.c | 30 ++++++++++++++++++++++++------ hw/virtio/vhost-user.c | 3 +-- include/hw/virtio/vhost-user.h | 1 - 3 files changed, 25 insertions(+), 9 deletions(-) diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c index 156ed6de30..9531d92e48 100644 --- a/hw/block/vhost-user-blk.c +++ b/hw/block/vhost-user-blk.c @@ -135,10 +135,7 @@ static bool vhost_user_blk_inflight_needed(void *opaque) { struct VHostUserBlk *s = opaque; - bool inflight_migration = virtio_has_feature(s->dev.protocol_features, - VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT); - - return inflight_migration; + return s->inflight_migration; } @@ -239,11 +236,14 @@ static int vhost_user_blk_stop(VirtIODevice *vdev) return 0; } + bool skip_drain = vhost_user_blk_inflight_needed(s) && + runstate_check(RUN_STATE_FINISH_MIGRATE); + force_stop = s->skip_get_vring_base_on_force_shutdown && qemu_force_shutdown_requested(); ret = force_stop ? vhost_dev_force_stop(&s->dev, vdev, true) : - vhost_dev_stop(&s->dev, vdev, true, false); + vhost_dev_stop(&s->dev, vdev, true, skip_drain); err = k->set_guest_notifiers(qbus->parent, s->dev.nvqs, false); if (err < 0) { @@ -375,7 +375,6 @@ static int vhost_user_blk_connect(DeviceState *dev, Error **errp) vhost_dev_set_config_notifier(&s->dev, &blk_ops); s->vhost_user.supports_config = true; - s->vhost_user.supports_inflight_migration = s->inflight_migration; ret = vhost_dev_init(&s->dev, &s->vhost_user, VHOST_BACKEND_TYPE_USER, 0, errp); if (ret < 0) { @@ -598,10 +597,29 @@ static struct vhost_dev *vhost_user_blk_get_vhost(VirtIODevice *vdev) return &s->dev; } +static bool vhost_user_blk_pre_save(void *opaque, Error **errp) +{ + VHostUserBlk *s = VHOST_USER_BLK(opaque); + + bool inflight_migration_enabled = vhost_user_has_protocol_feature(&s->dev, + VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN); + + if (vhost_user_blk_inflight_needed(s) && !inflight_migration_enabled) { + error_setg(errp, "can't migrate vhost-user-blk device: " + "backend doesn't support " + "VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN " + "protocol feature"); + return false; + } + + return true; +} + static const VMStateDescription vmstate_vhost_user_blk_inflight = { .name = "vhost-user-blk/inflight", .version_id = 1, .needed = vhost_user_blk_inflight_needed, + .pre_save_errp = vhost_user_blk_pre_save, .fields = (const VMStateField[]) { VMSTATE_VHOST_INFLIGHT_REGION(inflight, VHostUserBlk), VMSTATE_END_OF_LIST() diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index bf455daa44..03b8bc20f9 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -2571,8 +2571,7 @@ static int vhost_user_backend_init(struct vhost_dev *dev, void *opaque, } } - if (!u->user->supports_inflight_migration || - !virtio_has_feature(protocol_features, + if (!virtio_has_feature(protocol_features, VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD)) { protocol_features &= ~(1ULL << VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT); diff --git a/include/hw/virtio/vhost-user.h b/include/hw/virtio/vhost-user.h index 47c13f8677..78b4fba27f 100644 --- a/include/hw/virtio/vhost-user.h +++ b/include/hw/virtio/vhost-user.h @@ -73,7 +73,6 @@ typedef struct VhostUserState { GPtrArray *notifiers; int memory_slots; bool supports_config; - bool supports_inflight_migration; } VhostUserState; /** -- 2.34.1 ^ permalink raw reply related [flat|nested] 18+ messages in thread
* Re: [PATCH v5 0/6] vhost-user-blk: add compatibility with older qemu versions 2026-07-28 10:08 [PATCH v5 0/6] vhost-user-blk: add compatibility with older qemu versions Alexandr Moshkov ` (5 preceding siblings ...) 2026-07-28 10:08 ` [PATCH v5 6/6] vhost-user-blk: use GET_VRING_BASE_SKIP_DRAIN during inflight migration Alexandr Moshkov @ 2026-07-29 9:14 ` Michael S. Tsirkin 6 siblings, 0 replies; 18+ messages in thread From: Michael S. Tsirkin @ 2026-07-29 9:14 UTC (permalink / raw) To: Alexandr Moshkov Cc: qemu-devel, Hanna Reitz, Jason Wang, Vladimir Sementsov-Ogievskiy, Gonglei (Arei), Jason Wang, Fam Zheng, qemu-block, Stefan Hajnoczi, Alex Bennée, yc-core@yandex-team.ru, Pierrick Bouvier, Pierrick Bouvier, virtio-fs, Kevin Wolf, zhenwei pi, Paolo Bonzini, Stefano Garzarella, Milan Zamazal, Raphael Norwitz On Tue, Jul 28, 2026 at 03:08:35PM +0500, Alexandr Moshkov wrote: > v4 -> v5: > - introduce protocol feature VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN > to guard the new message, instead of reusing the existing > VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INLIFGHT. Update docs. > - improved commit messages. > > v3 -> v4: > - add new protocol message GET_VRING_BASE_SKIP_DRAIN instead of parameter to GET_VRING_BASE message. This new message can be send instead of GET_VRING_BASE, allowing back-end to suspend inflight I/O immediately instead of waiting and completing them. > - rebase to newer master > > v2 -> v3: > - fix complile problems > - add assert check in do_vhost_virtio_stop > - make inflight-migration property mutable > > v1 -> v2: > - reorganize commits: make refactor commits first, then core semantic change > - add additional pre_save check for inflight migration possibility > > --- > > This is v5 of the series previously sent as > "[PATCH v4 0/6] vhost-user-blk: fix inflight migration compatibility". > > This series extends the vhost-user-blk inflight migration feature > introduced in QEMU 11.0 to address a runtime compatibility problem. > > Currently, the inflight migration behaviour in vhost-user-blk is > controlled by VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT, which is > negotiated once at connection time. Once the feature is negotiated, the > back-end always uses suspend semantics on GET_VRING_BASE — there is no > way for the front-end to request normal drain behaviour on a per-call > basis without reconnecting. This makes it impossible to disable > inflight-migration at runtime, which is needed when, for example, > migrating a VM to an older QEMU version that does not support the > feature. This part, I do not understand. What exactly does "migrating" mean here? Live migration? An older QEMU will presumably either negotiate or fail to negotiate the bit. > To solve this, the series introduces a new protocol message > VHOST_USER_GET_VRING_BASE_SKIP_DRAIN (id=45) guarded by a new protocol > feature VHOST_USER_PROTOCOL_F_GET_VRING_BASE_SKIP_DRAIN. The message is > semantically identical to GET_VRING_BASE but explicitly instructs the > back-end to suspend in-flight I/O immediately rather than draining it. > This gives the front-end explicit per-call control: GET_VRING_BASE for > normal operation, GET_VRING_BASE_SKIP_DRAIN during live migration. > > In vhost-user-blk, GET_VRING_BASE_SKIP_DRAIN is sent only when > inflight-migration is enabled and the device is stopping due to live > migration (RUN_STATE_FINISH_MIGRATE). The inflight-migration property > is also made mutable so it can be toggled via qom-set at runtime without > reconnecting to the back-end. > > Alexandr Moshkov (6): > vhost-user: add skip_drain param to do_vhost_virtqueue_stop > vhost-user: add GET_VRING_BASE_SKIP_DRAIN message > vhost-user: use skip_drain with GET_VRING_BASE_SKIP_DRAIN message > vhost-user-blk: make inflight-migration prop mutable > vhost-user-blk: move inflight_needed higher > vhost-user-blk: use GET_VRING_BASE_SKIP_DRAIN during inflight > migration > > backends/cryptodev-vhost.c | 2 +- > backends/vhost-user.c | 2 +- > docs/interop/vhost-user.rst | 88 +++++++++++++++++++------------ > hw/block/vhost-user-blk.c | 43 ++++++++++++--- > hw/net/vhost_net.c | 9 ++-- > hw/scsi/vhost-scsi-common.c | 2 +- > hw/virtio/vdpa-dev.c | 2 +- > hw/virtio/vhost-user-base.c | 2 +- > hw/virtio/vhost-user-fs.c | 2 +- > hw/virtio/vhost-user-scmi.c | 2 +- > hw/virtio/vhost-user.c | 47 ++++++++++++++--- > hw/virtio/vhost-vsock-common.c | 2 +- > hw/virtio/vhost.c | 34 ++++++++---- > include/hw/virtio/vhost-backend.h | 1 + > include/hw/virtio/vhost-user.h | 2 +- > include/hw/virtio/vhost.h | 7 ++- > 16 files changed, 172 insertions(+), 75 deletions(-) > > -- > 2.34.1 ^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2026-07-29 18:50 UTC | newest]
Thread overview: 18+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-07-28 10:08 [PATCH v5 0/6] vhost-user-blk: add compatibility with older qemu versions Alexandr Moshkov
2026-07-28 10:08 ` [PATCH v5 1/6] vhost-user: add skip_drain param to do_vhost_virtqueue_stop Alexandr Moshkov
2026-07-28 10:08 ` [PATCH v5 2/6] vhost-user: add GET_VRING_BASE_SKIP_DRAIN message Alexandr Moshkov
2026-07-29 9:14 ` Michael S. Tsirkin
2026-07-28 10:08 ` [PATCH v5 3/6] vhost-user: use skip_drain with " Alexandr Moshkov
2026-07-28 10:08 ` [PATCH v5 4/6] vhost-user-blk: make inflight-migration prop mutable Alexandr Moshkov
2026-07-29 9:32 ` Michael S. Tsirkin
[not found] ` <679a33d1-0f6c-4456-ac76-1f9e1251f2ec@yandex-team.ru>
2026-07-29 10:14 ` Michael S. Tsirkin
2026-07-29 10:35 ` Alexandr Moshkov
2026-07-29 10:48 ` Michael S. Tsirkin
[not found] ` <dd1f5830-75ff-4256-a179-6a8341a3b4ec@yandex-team.ru>
2026-07-29 12:13 ` Michael S. Tsirkin
2026-07-29 14:07 ` Alexandr Moshkov
2026-07-29 14:48 ` Michael S. Tsirkin
2026-07-29 15:18 ` Alexandr Moshkov
2026-07-29 18:50 ` Michael S. Tsirkin
2026-07-28 10:08 ` [PATCH v5 5/6] vhost-user-blk: move inflight_needed higher Alexandr Moshkov
2026-07-28 10:08 ` [PATCH v5 6/6] vhost-user-blk: use GET_VRING_BASE_SKIP_DRAIN during inflight migration Alexandr Moshkov
2026-07-29 9:14 ` [PATCH v5 0/6] vhost-user-blk: add compatibility with older qemu versions Michael S. Tsirkin
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox