[PATCH v6 0/5] support inflight migration

[PATCH v6 0/5] support inflight migration

[Alexandr Moshkov]

v6:
- fix documentation about new protocol feature
- add check to ensure that inflight buffer subsection has been successfully loaded
- disable support for the new feature if in-flight or inflight migration is not supported.

v5:
Make protocol feature flag instead of GET_VRING_BASE msg parameter,
so all changes in other devices is no longer needed.
Now back-end may set this feature for QEMU. This feature must be set
with in-flight migration parameter in vhost-user-blk.

v4:
While testing inflight migration, I notices a problem with the fact that
GET_VRING_BASE is needed during migration, so the back-end stops
dirtying pages and synchronizes `last_avail` counter with QEMU. So after
migration in-flight I/O requests will be looks like resubmited on destination vm.

However, in new logic, we no longer need to wait for in-flight requests
to be complete at GET_VRING_BASE message. So support new parameter
`should_drain` in the GET_VRING_BASE to allow back-end stop vrings
immediately without waiting for in-flight I/O requests to complete.

Also:
- modify vhost-user rst
- refactor on vhost-user-blk.c, now `should_drain` is based on
  device parameter `inflight-migration`

v3:
- use pre_load_errp instead of pre_load in vhost.c
- change vhost-user-blk property to
  "skip-get-vring-base-inflight-migration"
- refactor vhost-user-blk.c, by moving vhost_user_blk_inflight_needed() higher

v2:
- rewrite migration using VMSD instead of qemufile API
- add vhost-user-blk parameter instead of migration capability

I don't know if VMSD was used cleanly in migration implementation, so
feel free for comments.

Based on Vladimir's work:
[PATCH v2 00/25] vhost-user-blk: live-backend local migration
  which was based on:
    - [PATCH v4 0/7] chardev: postpone connect
      (which in turn is based on [PATCH 0/2] remove deprecated 'reconnect' options)
    - [PATCH v3 00/23] vhost refactoring and fixes
    - [PATCH v8 14/19] migration: introduce .pre_incoming() vmsd handler

Based-on: <20250924133309.334631-1-vsementsov@yandex-team.ru>
Based-on: <20251015212051.1156334-1-vsementsov@yandex-team.ru>
Based-on: <20251015145808.1112843-1-vsementsov@yandex-team.ru>
Based-on: <20251015132136.1083972-15-vsementsov@yandex-team.ru>
Based-on: <20251016114104.1384675-1-vsementsov@yandex-team.ru>

---

Hi!

During inter-host migration, waiting for disk requests to be drained
in the vhost-user backend can incur significant downtime.

This can be avoided if QEMU migrates the inflight region in
vhost-user-blk.
Thus, during the qemu migration, with protocol feature flag the vhost-user
back-end can immediately stop vrings, so all in-flight requests will be
migrated to another host.

At first, I tried to implement migration for all vhost-user devices that support inflight at once,
but this would require a lot of changes both in vhost-user-blk (to transfer it to the base class) and
in the vhost-user-base base class (inflight implementation and remodeling + a large refactor).

Therefore, for now I decided to leave this idea for later and
implement the migration of the inflight region first for vhost-user-blk.

Alexandr Moshkov (5):
  vhost-user.rst: specify vhost-user back-end action on GET_VRING_BASE
  vhost-user: introduce protocol feature for skip drain on
    GET_VRING_BASE
  vmstate: introduce VMSTATE_VBUFFER_UINT64
  vhost: add vmstate for inflight region with inner buffer
  vhost-user-blk: support inter-host inflight migration

 docs/interop/vhost-user.rst        | 56 ++++++++++++++++++-----------
 hw/block/vhost-user-blk.c          | 28 +++++++++++++++
 hw/virtio/vhost-user.c             |  5 +++
 hw/virtio/vhost.c                  | 57 ++++++++++++++++++++++++++++++
 include/hw/virtio/vhost-user-blk.h |  1 +
 include/hw/virtio/vhost-user.h     |  2 ++
 include/hw/virtio/vhost.h          |  6 ++++
 include/migration/vmstate.h        | 10 ++++++
 8 files changed, 145 insertions(+), 20 deletions(-)

-- 
2.34.1

[PATCH v6 1/5] vhost-user.rst: specify vhost-user back-end action on GET_VRING_BASE

[Alexandr Moshkov]

By default, we assume that server need to wait all inflight IO on
GET_VRING_BASE. However, this fact is not recorded anywhere in the
documentation.
So, add this info in rst.

Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
---
 docs/interop/vhost-user.rst | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
index 2e50f2ddfa..02908b48fa 100644
--- a/docs/interop/vhost-user.rst
+++ b/docs/interop/vhost-user.rst
@@ -1243,7 +1243,8 @@ Front-end message types
 
   When and as long as all of a device's vrings are stopped, it is
   *suspended*, see :ref:`Suspended device state
-  <suspended_device_state>`.
+  <suspended_device_state>`. The back-end must complete all inflight I/O
+  requests for the specified vring before stopping it.
 
   The request payload's *num* field is currently reserved and must be
   set to 0.
-- 
2.34.1

[PATCH v6 2/5] vhost-user: introduce protocol feature for skip drain on GET_VRING_BASE

[Alexandr Moshkov]

Add vhost-user protocol feature
VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT

Now on GET_VRING_BASE this feature can control whether to wait for
in-flight requests to complete or not.
Also we have to validate that this feature will be enabled only when
qemu and back-end supports in-flight buffer and in-flight migration

It will be helpfull in future for in-flight requests migration in
vhost-user devices.

Update docs, add ref to label for inflight-io-tracking

Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
---
 docs/interop/vhost-user.rst    | 59 +++++++++++++++++++++-------------
 hw/virtio/vhost-user.c         |  5 +++
 include/hw/virtio/vhost-user.h |  2 ++
 3 files changed, 44 insertions(+), 22 deletions(-)

diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
index 02908b48fa..dcf79de0c0 100644
--- a/docs/interop/vhost-user.rst
+++ b/docs/interop/vhost-user.rst
@@ -736,6 +736,8 @@ negotiated, back-end can send file descriptors (at most 8 descriptors in
 each message) to front-end via ancillary data using this fd communication
 channel.
 
+.. _inflight_io_tracking:
+
 Inflight I/O tracking
 ---------------------
 
@@ -1033,26 +1035,27 @@ Protocol features
 
 .. code:: c
 
-  #define VHOST_USER_PROTOCOL_F_MQ                    0
-  #define VHOST_USER_PROTOCOL_F_LOG_SHMFD             1
-  #define VHOST_USER_PROTOCOL_F_RARP                  2
-  #define VHOST_USER_PROTOCOL_F_REPLY_ACK             3
-  #define VHOST_USER_PROTOCOL_F_MTU                   4
-  #define VHOST_USER_PROTOCOL_F_BACKEND_REQ           5
-  #define VHOST_USER_PROTOCOL_F_CROSS_ENDIAN          6
-  #define VHOST_USER_PROTOCOL_F_CRYPTO_SESSION        7
-  #define VHOST_USER_PROTOCOL_F_PAGEFAULT             8
-  #define VHOST_USER_PROTOCOL_F_CONFIG                9
-  #define VHOST_USER_PROTOCOL_F_BACKEND_SEND_FD      10
-  #define VHOST_USER_PROTOCOL_F_HOST_NOTIFIER        11
-  #define VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD       12
-  #define VHOST_USER_PROTOCOL_F_RESET_DEVICE         13
-  #define VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS 14
-  #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS  15
-  #define VHOST_USER_PROTOCOL_F_STATUS               16
-  #define VHOST_USER_PROTOCOL_F_XEN_MMAP             17
-  #define VHOST_USER_PROTOCOL_F_SHARED_OBJECT        18
-  #define VHOST_USER_PROTOCOL_F_DEVICE_STATE         19
+  #define VHOST_USER_PROTOCOL_F_MQ                       0
+  #define VHOST_USER_PROTOCOL_F_LOG_SHMFD                1
+  #define VHOST_USER_PROTOCOL_F_RARP                     2
+  #define VHOST_USER_PROTOCOL_F_REPLY_ACK                3
+  #define VHOST_USER_PROTOCOL_F_MTU                      4
+  #define VHOST_USER_PROTOCOL_F_BACKEND_REQ              5
+  #define VHOST_USER_PROTOCOL_F_CROSS_ENDIAN             6
+  #define VHOST_USER_PROTOCOL_F_CRYPTO_SESSION           7
+  #define VHOST_USER_PROTOCOL_F_PAGEFAULT                8
+  #define VHOST_USER_PROTOCOL_F_CONFIG                   9
+  #define VHOST_USER_PROTOCOL_F_BACKEND_SEND_FD         10
+  #define VHOST_USER_PROTOCOL_F_HOST_NOTIFIER           11
+  #define VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD          12
+  #define VHOST_USER_PROTOCOL_F_RESET_DEVICE            13
+  #define VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS    14
+  #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS     15
+  #define VHOST_USER_PROTOCOL_F_STATUS                  16
+  #define VHOST_USER_PROTOCOL_F_XEN_MMAP                17
+  #define VHOST_USER_PROTOCOL_F_SHARED_OBJECT           18
+  #define VHOST_USER_PROTOCOL_F_DEVICE_STATE            19
+  #define VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT 20
 
 Front-end message types
 -----------------------
@@ -1243,12 +1246,24 @@ Front-end message types
 
   When and as long as all of a device's vrings are stopped, it is
   *suspended*, see :ref:`Suspended device state
-  <suspended_device_state>`. The back-end must complete all inflight I/O
-  requests for the specified vring before stopping it.
+  <suspended_device_state>`.
 
   The request payload's *num* field is currently reserved and must be
   set to 0.
 
+  By default, the back-end must complete all inflight I/O requests for the
+  specified vring before stopping it.
+
+  If the ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT`` protocol
+  feature has been negotiated, the back-end may suspend in-flight I/O
+  requests and record them as described in :ref:`Inflight I/O tracking
+  <inflight_io_tracking>` instead of completing them before stopping the vring.
+  How to suspend an in-flight request depends on the implementation of the back-end
+  but it typically can be done by aborting or cancelling the underlying I/O
+  request. The ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT``
+  protocol feature must only be neogotiated if
+  ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD`` is also negotiated.
+
 ``VHOST_USER_SET_VRING_KICK``
   :id: 12
   :equivalent ioctl: ``VHOST_SET_VRING_KICK``
diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c
index a820214188..793d1b36d8 100644
--- a/hw/virtio/vhost-user.c
+++ b/hw/virtio/vhost-user.c
@@ -2320,6 +2320,11 @@ static int vhost_user_backend_connect(struct vhost_dev *dev, Error **errp)
             }
         }
 
+        if (!u->user->supports_inflight_migration ||
+            !virtio_has_feature(VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD)) {
+            protocol_features &= ~(1ULL <<
+                               VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
+        }
+
         /* final set of protocol features */
         u->protocol_features = protocol_features;
         err = vhost_user_set_protocol_features(dev, u->protocol_features);
diff --git a/include/hw/virtio/vhost-user.h b/include/hw/virtio/vhost-user.h
index fb89268de2..f30c1792e4 100644
--- a/include/hw/virtio/vhost-user.h
+++ b/include/hw/virtio/vhost-user.h
@@ -33,6 +33,7 @@ enum VhostUserProtocolFeature {
     /* Feature 17 reserved for VHOST_USER_PROTOCOL_F_XEN_MMAP. */
     VHOST_USER_PROTOCOL_F_SHARED_OBJECT = 18,
     VHOST_USER_PROTOCOL_F_DEVICE_STATE = 19,
+    VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT = 20,
     VHOST_USER_PROTOCOL_F_MAX
 };
 
@@ -69,6 +70,7 @@ typedef struct VhostUserState {
     GPtrArray *notifiers;
     int memory_slots;
     bool supports_config;
+    bool supports_inflight_migration;
 } VhostUserState;
 
 /**
-- 
2.34.1

[PATCH v6 3/5] vmstate: introduce VMSTATE_VBUFFER_UINT64

[Alexandr Moshkov]

This is an analog of VMSTATE_VBUFFER_UINT32 macro, but for uint64 type.

Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
Acked-by: Peter Xu <peterx@redhat.com>
---
 include/migration/vmstate.h | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h
index 7f1f1c166a..4c9e212d58 100644
--- a/include/migration/vmstate.h
+++ b/include/migration/vmstate.h
@@ -707,6 +707,16 @@ extern const VMStateInfo vmstate_info_qlist;
     .offset       = offsetof(_state, _field),                        \
 }
 
+#define VMSTATE_VBUFFER_UINT64(_field, _state, _version, _test, _field_size) { \
+    .name         = (stringify(_field)),                             \
+    .version_id   = (_version),                                      \
+    .field_exists = (_test),                                         \
+    .size_offset  = vmstate_offset_value(_state, _field_size, uint64_t),\
+    .info         = &vmstate_info_buffer,                            \
+    .flags        = VMS_VBUFFER | VMS_POINTER,                       \
+    .offset       = offsetof(_state, _field),                        \
+}
+
 #define VMSTATE_VBUFFER_ALLOC_UINT32(_field, _state, _version,       \
                                      _test, _field_size) {           \
     .name         = (stringify(_field)),                             \
-- 
2.34.1

[PATCH v6 4/5] vhost: add vmstate for inflight region with inner buffer

[Alexandr Moshkov]

Prepare for future inflight region migration for vhost-user-blk.
We need to migrate size, queue_size, and inner buffer.

So firstly it migrate size and queue_size fields, then allocate memory for buffer with
migrated size, then migrate inner buffer itself.

Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
---
 hw/virtio/vhost.c         | 57 +++++++++++++++++++++++++++++++++++++++
 include/hw/virtio/vhost.h |  6 +++++
 2 files changed, 63 insertions(+)

diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index c46203eb9c..f655c53b67 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -2028,6 +2028,63 @@ const VMStateDescription vmstate_backend_transfer_vhost_inflight = {
     }
 };
 
+static int vhost_inflight_buffer_pre_load(void *opaque, Error **errp)
+{
+    info_report("vhost_inflight_region_buffer_pre_load");
+    struct vhost_inflight *inflight = opaque;
+
+    int fd = -1;
+    void *addr = qemu_memfd_alloc("vhost-inflight", inflight->size,
+                                  F_SEAL_GROW | F_SEAL_SHRINK | F_SEAL_SEAL,
+                                  &fd, errp);
+    if (*errp) {
+        return -ENOMEM;
+    }
+
+    inflight->offset = 0;
+    inflight->addr = addr;
+    inflight->fd = fd;
+
+    return 0;
+}
+
+const VMStateDescription vmstate_vhost_inflight_region_buffer = {
+    .name = "vhost-inflight-region/buffer",
+    .pre_load_errp = vhost_inflight_buffer_pre_load,
+    .fields = (const VMStateField[]) {
+        VMSTATE_VBUFFER_UINT64(addr, struct vhost_inflight, 0, NULL, size),
+        VMSTATE_END_OF_LIST()
+    }
+};
+
+static int vhost_inflight_region_post_load(void *opaque,
+                                           int version_id,
+                                           Error **errp)
+{
+    struct vhost_inflight *inflight = opaque;
+
+    if (inflight->addr == NULL) {
+        error_setg(errp, "inflight buffer subsection has not been loaded");
+        return -EINVAL;
+    }
+
+    return 0;
+}
+
+const VMStateDescription vmstate_vhost_inflight_region = {
+    .name = "vhost-inflight-region",
+    .post_load_errp = vhost_inflight_region_post_load,
+    .fields = (const VMStateField[]) {
+        VMSTATE_UINT64(size, struct vhost_inflight),
+        VMSTATE_UINT16(queue_size, struct vhost_inflight),
+        VMSTATE_END_OF_LIST()
+    },
+    .subsections = (const VMStateDescription * const []) {
+        &vmstate_vhost_inflight_region_buffer,
+        NULL
+    }
+};
+
 const VMStateDescription vmstate_vhost_virtqueue = {
     .name = "vhost-virtqueue",
     .fields = (const VMStateField[]) {
diff --git a/include/hw/virtio/vhost.h b/include/hw/virtio/vhost.h
index 13ca2c319f..dd552de91f 100644
--- a/include/hw/virtio/vhost.h
+++ b/include/hw/virtio/vhost.h
@@ -596,6 +596,12 @@ extern const VMStateDescription vmstate_backend_transfer_vhost_inflight;
                            vmstate_backend_transfer_vhost_inflight, \
                            struct vhost_inflight)
 
+extern const VMStateDescription vmstate_vhost_inflight_region;
+#define VMSTATE_VHOST_INFLIGHT_REGION(_field, _state) \
+    VMSTATE_STRUCT_POINTER(_field, _state, \
+                           vmstate_vhost_inflight_region, \
+                           struct vhost_inflight)
+
 extern const VMStateDescription vmstate_vhost_dev;
 #define VMSTATE_BACKEND_TRANSFER_VHOST(_field, _state) \
     VMSTATE_STRUCT(_field, _state, 0, vmstate_vhost_dev, struct vhost_dev)
-- 
2.34.1

[PATCH v6 5/5] vhost-user-blk: support inter-host inflight migration

[Alexandr Moshkov]

During inter-host migration, waiting for disk requests to be drained
in the vhost-user backend can incur significant downtime.

This can be avoided if QEMU migrates the inflight region in
vhost-user-blk.
Thus, during the qemu migration, with feature flag the vhost-user
back-end can immediately stop vrings, so all in-flight requests will be
migrated to another host.

Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
---
 hw/block/vhost-user-blk.c          | 28 ++++++++++++++++++++++++++++
 include/hw/virtio/vhost-user-blk.h |  1 +
 2 files changed, 29 insertions(+)

diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
index a8fd90480a..9093e98841 100644
--- a/hw/block/vhost-user-blk.c
+++ b/hw/block/vhost-user-blk.c
@@ -377,6 +377,7 @@ static int vhost_user_blk_init(DeviceState *dev, bool connect, Error **errp)
     vhost_dev_set_config_notifier(&s->dev, &blk_ops);
 
     s->vhost_user.supports_config = true;
+    s->vhost_user.supports_inflight_migration = s->inflight_migration;
     ret = vhost_dev_init(&s->dev, &s->vhost_user, VHOST_BACKEND_TYPE_USER, 0,
                          false, errp);
     if (ret < 0) {
@@ -656,6 +657,27 @@ static struct vhost_dev *vhost_user_blk_get_vhost(VirtIODevice *vdev)
     return &s->dev;
 }
 
+static bool vhost_user_blk_inflight_needed(void *opaque)
+{
+    struct VHostUserBlk *s = opaque;
+
+    bool inflight_migration = vhost_dev_has_feature(&s->dev,
+                        VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
+
+    return inflight_migration &&
+           !migrate_local_vhost_user_blk();
+}
+
+static const VMStateDescription vmstate_vhost_user_blk_inflight = {
+    .name = "vhost-user-blk/inflight",
+    .version_id = 1,
+    .needed = vhost_user_blk_inflight_needed,
+    .fields = (const VMStateField[]) {
+        VMSTATE_VHOST_INFLIGHT_REGION(inflight, VHostUserBlk),
+        VMSTATE_END_OF_LIST()
+    },
+};
+
 static bool vhost_user_blk_pre_incoming(void *opaque, Error **errp)
 {
     VHostUserBlk *s = VHOST_USER_BLK(opaque);
@@ -678,6 +700,10 @@ static const VMStateDescription vmstate_vhost_user_blk = {
         VMSTATE_VIRTIO_DEVICE,
         VMSTATE_END_OF_LIST()
     },
+    .subsections = (const VMStateDescription * const []) {
+        &vmstate_vhost_user_blk_inflight,
+        NULL
+    }
 };
 
 static bool vhost_user_needed(void *opaque)
@@ -751,6 +777,8 @@ static const Property vhost_user_blk_properties[] = {
                       VIRTIO_BLK_F_WRITE_ZEROES, true),
     DEFINE_PROP_BOOL("skip-get-vring-base-on-force-shutdown", VHostUserBlk,
                      skip_get_vring_base_on_force_shutdown, false),
+    DEFINE_PROP_BOOL("inflight-migration", VHostUserBlk,
+                     inflight_migration, false),
 };
 
 static void vhost_user_blk_class_init(ObjectClass *klass, const void *data)
diff --git a/include/hw/virtio/vhost-user-blk.h b/include/hw/virtio/vhost-user-blk.h
index b06f55fd6f..e1466e5cf6 100644
--- a/include/hw/virtio/vhost-user-blk.h
+++ b/include/hw/virtio/vhost-user-blk.h
@@ -52,6 +52,7 @@ struct VHostUserBlk {
     bool started_vu;
 
     bool skip_get_vring_base_on_force_shutdown;
+    bool inflight_migration;
 
     bool incoming_backend;
 };
-- 
2.34.1

Re: [PATCH v6 2/5] vhost-user: introduce protocol feature for skip drain on GET_VRING_BASE

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 7189 bytes --]

On Tue, Jan 13, 2026 at 02:58:13PM +0500, Alexandr Moshkov wrote:
> Add vhost-user protocol feature
> VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT
> 
> Now on GET_VRING_BASE this feature can control whether to wait for
> in-flight requests to complete or not.
> Also we have to validate that this feature will be enabled only when
> qemu and back-end supports in-flight buffer and in-flight migration
> 
> It will be helpfull in future for in-flight requests migration in
> vhost-user devices.
> 
> Update docs, add ref to label for inflight-io-tracking
> 
> Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
> ---
>  docs/interop/vhost-user.rst    | 59 +++++++++++++++++++++-------------
>  hw/virtio/vhost-user.c         |  5 +++
>  include/hw/virtio/vhost-user.h |  2 ++
>  3 files changed, 44 insertions(+), 22 deletions(-)
> 
> diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
> index 02908b48fa..dcf79de0c0 100644
> --- a/docs/interop/vhost-user.rst
> +++ b/docs/interop/vhost-user.rst
> @@ -736,6 +736,8 @@ negotiated, back-end can send file descriptors (at most 8 descriptors in
>  each message) to front-end via ancillary data using this fd communication
>  channel.
>  
> +.. _inflight_io_tracking:
> +
>  Inflight I/O tracking
>  ---------------------
>  
> @@ -1033,26 +1035,27 @@ Protocol features
>  
>  .. code:: c
>  
> -  #define VHOST_USER_PROTOCOL_F_MQ                    0
> -  #define VHOST_USER_PROTOCOL_F_LOG_SHMFD             1
> -  #define VHOST_USER_PROTOCOL_F_RARP                  2
> -  #define VHOST_USER_PROTOCOL_F_REPLY_ACK             3
> -  #define VHOST_USER_PROTOCOL_F_MTU                   4
> -  #define VHOST_USER_PROTOCOL_F_BACKEND_REQ           5
> -  #define VHOST_USER_PROTOCOL_F_CROSS_ENDIAN          6
> -  #define VHOST_USER_PROTOCOL_F_CRYPTO_SESSION        7
> -  #define VHOST_USER_PROTOCOL_F_PAGEFAULT             8
> -  #define VHOST_USER_PROTOCOL_F_CONFIG                9
> -  #define VHOST_USER_PROTOCOL_F_BACKEND_SEND_FD      10
> -  #define VHOST_USER_PROTOCOL_F_HOST_NOTIFIER        11
> -  #define VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD       12
> -  #define VHOST_USER_PROTOCOL_F_RESET_DEVICE         13
> -  #define VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS 14
> -  #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS  15
> -  #define VHOST_USER_PROTOCOL_F_STATUS               16
> -  #define VHOST_USER_PROTOCOL_F_XEN_MMAP             17
> -  #define VHOST_USER_PROTOCOL_F_SHARED_OBJECT        18
> -  #define VHOST_USER_PROTOCOL_F_DEVICE_STATE         19
> +  #define VHOST_USER_PROTOCOL_F_MQ                       0
> +  #define VHOST_USER_PROTOCOL_F_LOG_SHMFD                1
> +  #define VHOST_USER_PROTOCOL_F_RARP                     2
> +  #define VHOST_USER_PROTOCOL_F_REPLY_ACK                3
> +  #define VHOST_USER_PROTOCOL_F_MTU                      4
> +  #define VHOST_USER_PROTOCOL_F_BACKEND_REQ              5
> +  #define VHOST_USER_PROTOCOL_F_CROSS_ENDIAN             6
> +  #define VHOST_USER_PROTOCOL_F_CRYPTO_SESSION           7
> +  #define VHOST_USER_PROTOCOL_F_PAGEFAULT                8
> +  #define VHOST_USER_PROTOCOL_F_CONFIG                   9
> +  #define VHOST_USER_PROTOCOL_F_BACKEND_SEND_FD         10
> +  #define VHOST_USER_PROTOCOL_F_HOST_NOTIFIER           11
> +  #define VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD          12
> +  #define VHOST_USER_PROTOCOL_F_RESET_DEVICE            13
> +  #define VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS    14
> +  #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS     15
> +  #define VHOST_USER_PROTOCOL_F_STATUS                  16
> +  #define VHOST_USER_PROTOCOL_F_XEN_MMAP                17
> +  #define VHOST_USER_PROTOCOL_F_SHARED_OBJECT           18
> +  #define VHOST_USER_PROTOCOL_F_DEVICE_STATE            19
> +  #define VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT 20
>  
>  Front-end message types
>  -----------------------
> @@ -1243,12 +1246,24 @@ Front-end message types
>  
>    When and as long as all of a device's vrings are stopped, it is
>    *suspended*, see :ref:`Suspended device state
> -  <suspended_device_state>`. The back-end must complete all inflight I/O
> -  requests for the specified vring before stopping it.
> +  <suspended_device_state>`.
>  
>    The request payload's *num* field is currently reserved and must be
>    set to 0.
>  
> +  By default, the back-end must complete all inflight I/O requests for the
> +  specified vring before stopping it.
> +
> +  If the ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT`` protocol
> +  feature has been negotiated, the back-end may suspend in-flight I/O
> +  requests and record them as described in :ref:`Inflight I/O tracking
> +  <inflight_io_tracking>` instead of completing them before stopping the vring.
> +  How to suspend an in-flight request depends on the implementation of the back-end
> +  but it typically can be done by aborting or cancelling the underlying I/O
> +  request. The ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT``
> +  protocol feature must only be neogotiated if
> +  ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD`` is also negotiated.
> +
>  ``VHOST_USER_SET_VRING_KICK``
>    :id: 12
>    :equivalent ioctl: ``VHOST_SET_VRING_KICK``
> diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c
> index a820214188..793d1b36d8 100644
> --- a/hw/virtio/vhost-user.c
> +++ b/hw/virtio/vhost-user.c
> @@ -2320,6 +2320,11 @@ static int vhost_user_backend_connect(struct vhost_dev *dev, Error **errp)
>              }
>          }
>  
> +        if (!u->user->supports_inflight_migration ||
> +            !virtio_has_feature(VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD)) {

This looks like it will not compile since virtio_has_features() takes
two arguments. I think it should be:

  !virtio_has_feature(dev->protocol_features,
                      VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD)

> +            protocol_features &= ~(1ULL <<
> +                               VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
> +        }
> +
>          /* final set of protocol features */
>          u->protocol_features = protocol_features;
>          err = vhost_user_set_protocol_features(dev, u->protocol_features);

I seem to have different version of hw/virtio/vhost-user.c. Is this
patch against qemu.git/master?

> diff --git a/include/hw/virtio/vhost-user.h b/include/hw/virtio/vhost-user.h
> index fb89268de2..f30c1792e4 100644
> --- a/include/hw/virtio/vhost-user.h
> +++ b/include/hw/virtio/vhost-user.h
> @@ -33,6 +33,7 @@ enum VhostUserProtocolFeature {
>      /* Feature 17 reserved for VHOST_USER_PROTOCOL_F_XEN_MMAP. */
>      VHOST_USER_PROTOCOL_F_SHARED_OBJECT = 18,
>      VHOST_USER_PROTOCOL_F_DEVICE_STATE = 19,
> +    VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT = 20,
>      VHOST_USER_PROTOCOL_F_MAX
>  };
>  
> @@ -69,6 +70,7 @@ typedef struct VhostUserState {
>      GPtrArray *notifiers;
>      int memory_slots;
>      bool supports_config;
> +    bool supports_inflight_migration;
>  } VhostUserState;
>  
>  /**
> -- 
> 2.34.1
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v6 0/5] support inflight migration

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 542 bytes --]

On Tue, Jan 13, 2026 at 02:58:09PM +0500, Alexandr Moshkov wrote:

Peter: Please review the migration aspects (especially the vmstates).
Thank you!

> v6:
> - fix documentation about new protocol feature
> - add check to ensure that inflight buffer subsection has been successfully loaded
> - disable support for the new feature if in-flight or inflight migration is not supported.

Aside from the comment I posted:

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

This patch series should go through Michael Tsirkin's VIRTIO/vhost tree.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v6 0/5] support inflight migration

[Peter Xu]

On Tue, Jan 13, 2026 at 01:12:42PM -0500, Stefan Hajnoczi wrote:
> On Tue, Jan 13, 2026 at 02:58:09PM +0500, Alexandr Moshkov wrote:
> 
> Peter: Please review the migration aspects (especially the vmstates).
> Thank you!

Looks good from my side as long as it's based on VMSD, I appreciate that
change from the old versions where it used to use qemufile APIs.

The major question here is if this series depends on Vladimir's other
series, while there's still one patch that is not-for-merge:

https://lore.kernel.org/all/20251016114104.1384675-24-vsementsov@yandex-team.ru/#t

Does it automatically mark this series RFC as well?

Personally speaking, a new migration cap would work all fine, we should
have discussed it somewhere previously.  Said that, "local-vhost-user-blk"
capability is likely not the right one.  IMHO it should be either "local"
or "fd-passing" / "fd-passthrough" (or something generic) as the name.  If
we are not sure if we will leverage more than "passing the FDs around", we
can make it as simple as "local" as a new migration capability.

Then migration's misc.h should export a function migrate_is_local() then
device code can probe that in its own vmstate handling paths on save/load.

A note to Vladimir: please remember to add a check to enforce UNIX socket
when a formal patch 23 will be proposed some day, no matter what is the
name of the capability.  It should fail qmp "migrate" or qmp
"migrate_incoming" command if the main URI is not a unix socket.

Thanks,

-- 
Peter Xu

Re: [PATCH v6 0/5] support inflight migration

[Vladimir Sementsov-Ogievskiy]

On 13.01.26 21:56, Peter Xu wrote:
> On Tue, Jan 13, 2026 at 01:12:42PM -0500, Stefan Hajnoczi wrote:
>> On Tue, Jan 13, 2026 at 02:58:09PM +0500, Alexandr Moshkov wrote:
>>
>> Peter: Please review the migration aspects (especially the vmstates).
>> Thank you!
> 
> Looks good from my side as long as it's based on VMSD, I appreciate that
> change from the old versions where it used to use qemufile APIs.
> 
> The major question here is if this series depends on Vladimir's other
> series

No, it does not. And if we can proceed with merging these series first, I'll
be happy to rebase on top of it.

> while there's still one patch that is not-for-merge:
> 
> https://lore.kernel.org/all/20251016114104.1384675-24-vsementsov@yandex-team.ru/#t
> 
> Does it automatically mark this series RFC as well?
> 
> Personally speaking, a new migration cap would work all fine, we should
> have discussed it somewhere previously.  Said that, "local-vhost-user-blk"
> capability is likely not the right one.  IMHO it should be either "local"
> or "fd-passing" / "fd-passthrough" (or something generic) as the name.  If
> we are not sure if we will leverage more than "passing the FDs around", we
> can make it as simple as "local" as a new migration capability.
> 
> Then migration's misc.h should export a function migrate_is_local() then
> device code can probe that in its own vmstate handling paths on save/load.
> 
> A note to Vladimir: please remember to add a check to enforce UNIX socket
> when a formal patch 23 will be proposed some day, no matter what is the
> name of the capability.  It should fail qmp "migrate" or qmp
> "migrate_incoming" command if the main URI is not a unix socket.
> 

Thanks! I'll keep that in mind when prepare next version.

-- 
Best regards,
Vladimir

Re: [PATCH v6 0/5] support inflight migration

[Peter Xu]

On Wed, Jan 14, 2026 at 09:19:10AM +0300, Vladimir Sementsov-Ogievskiy wrote:
> On 13.01.26 21:56, Peter Xu wrote:
> > On Tue, Jan 13, 2026 at 01:12:42PM -0500, Stefan Hajnoczi wrote:
> > > On Tue, Jan 13, 2026 at 02:58:09PM +0500, Alexandr Moshkov wrote:
> > > 
> > > Peter: Please review the migration aspects (especially the vmstates).
> > > Thank you!
> > 
> > Looks good from my side as long as it's based on VMSD, I appreciate that
> > change from the old versions where it used to use qemufile APIs.
> > 
> > The major question here is if this series depends on Vladimir's other
> > series
> 
> No, it does not. And if we can proceed with merging these series first, I'll
> be happy to rebase on top of it.

I thought it requires migrate_local_vhost_user_blk() be present?  The
inflight feature should not be enabled only if there's a hint that it's a
local migration..

I'll comment inline on the patch later.

> 
> > while there's still one patch that is not-for-merge:
> > 
> > https://lore.kernel.org/all/20251016114104.1384675-24-vsementsov@yandex-team.ru/#t
> > 
> > Does it automatically mark this series RFC as well?
> > 
> > Personally speaking, a new migration cap would work all fine, we should
> > have discussed it somewhere previously.  Said that, "local-vhost-user-blk"
> > capability is likely not the right one.  IMHO it should be either "local"
> > or "fd-passing" / "fd-passthrough" (or something generic) as the name.  If
> > we are not sure if we will leverage more than "passing the FDs around", we
> > can make it as simple as "local" as a new migration capability.
> > 
> > Then migration's misc.h should export a function migrate_is_local() then
> > device code can probe that in its own vmstate handling paths on save/load.
> > 
> > A note to Vladimir: please remember to add a check to enforce UNIX socket
> > when a formal patch 23 will be proposed some day, no matter what is the
> > name of the capability.  It should fail qmp "migrate" or qmp
> > "migrate_incoming" command if the main URI is not a unix socket.
> > 
> 
> Thanks! I'll keep that in mind when prepare next version.
> 
> -- 
> Best regards,
> Vladimir
> 

-- 
Peter Xu

Re: [PATCH v6 5/5] vhost-user-blk: support inter-host inflight migration

[Peter Xu]

On Tue, Jan 13, 2026 at 02:58:19PM +0500, Alexandr Moshkov wrote:
> During inter-host migration, waiting for disk requests to be drained
> in the vhost-user backend can incur significant downtime.
> 
> This can be avoided if QEMU migrates the inflight region in
> vhost-user-blk.
> Thus, during the qemu migration, with feature flag the vhost-user
> back-end can immediately stop vrings, so all in-flight requests will be
> migrated to another host.
> 
> Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
> ---
>  hw/block/vhost-user-blk.c          | 28 ++++++++++++++++++++++++++++
>  include/hw/virtio/vhost-user-blk.h |  1 +
>  2 files changed, 29 insertions(+)
> 
> diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c
> index a8fd90480a..9093e98841 100644
> --- a/hw/block/vhost-user-blk.c
> +++ b/hw/block/vhost-user-blk.c
> @@ -377,6 +377,7 @@ static int vhost_user_blk_init(DeviceState *dev, bool connect, Error **errp)
>      vhost_dev_set_config_notifier(&s->dev, &blk_ops);
>  
>      s->vhost_user.supports_config = true;
> +    s->vhost_user.supports_inflight_migration = s->inflight_migration;
>      ret = vhost_dev_init(&s->dev, &s->vhost_user, VHOST_BACKEND_TYPE_USER, 0,
>                           false, errp);
>      if (ret < 0) {
> @@ -656,6 +657,27 @@ static struct vhost_dev *vhost_user_blk_get_vhost(VirtIODevice *vdev)
>      return &s->dev;
>  }
>  
> +static bool vhost_user_blk_inflight_needed(void *opaque)
> +{
> +    struct VHostUserBlk *s = opaque;
> +
> +    bool inflight_migration = vhost_dev_has_feature(&s->dev,
> +                        VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT);
> +
> +    return inflight_migration &&
> +           !migrate_local_vhost_user_blk();

Here's the spot that should depend on migrate_local_vhost_user_blk() from
Vladimilr's RFC patch (again, likely to be renamed..).

Btw, is this check correct against "!migrate_local_vhost_user_blk()"?  I
was expecting the feature off only if local=on, so I expect it to be:

       return inflight_migration && migrate_local_vhost_user_blk();

?

> +}
> +
> +static const VMStateDescription vmstate_vhost_user_blk_inflight = {
> +    .name = "vhost-user-blk/inflight",
> +    .version_id = 1,
> +    .needed = vhost_user_blk_inflight_needed,
> +    .fields = (const VMStateField[]) {
> +        VMSTATE_VHOST_INFLIGHT_REGION(inflight, VHostUserBlk),

One other trivial nitpick while glimpsing over the patch: should we move
the macro definition from previous patch to this one, where it is used?

> +        VMSTATE_END_OF_LIST()
> +    },
> +};
> +
>  static bool vhost_user_blk_pre_incoming(void *opaque, Error **errp)
>  {
>      VHostUserBlk *s = VHOST_USER_BLK(opaque);
> @@ -678,6 +700,10 @@ static const VMStateDescription vmstate_vhost_user_blk = {
>          VMSTATE_VIRTIO_DEVICE,
>          VMSTATE_END_OF_LIST()
>      },
> +    .subsections = (const VMStateDescription * const []) {
> +        &vmstate_vhost_user_blk_inflight,
> +        NULL
> +    }
>  };
>  
>  static bool vhost_user_needed(void *opaque)
> @@ -751,6 +777,8 @@ static const Property vhost_user_blk_properties[] = {
>                        VIRTIO_BLK_F_WRITE_ZEROES, true),
>      DEFINE_PROP_BOOL("skip-get-vring-base-on-force-shutdown", VHostUserBlk,
>                       skip_get_vring_base_on_force_shutdown, false),
> +    DEFINE_PROP_BOOL("inflight-migration", VHostUserBlk,
> +                     inflight_migration, false),
>  };
>  
>  static void vhost_user_blk_class_init(ObjectClass *klass, const void *data)
> diff --git a/include/hw/virtio/vhost-user-blk.h b/include/hw/virtio/vhost-user-blk.h
> index b06f55fd6f..e1466e5cf6 100644
> --- a/include/hw/virtio/vhost-user-blk.h
> +++ b/include/hw/virtio/vhost-user-blk.h
> @@ -52,6 +52,7 @@ struct VHostUserBlk {
>      bool started_vu;
>  
>      bool skip_get_vring_base_on_force_shutdown;
> +    bool inflight_migration;
>  
>      bool incoming_backend;
>  };
> -- 
> 2.34.1
> 

-- 
Peter Xu

Re: [PATCH v6 0/5] support inflight migration

[Vladimir Sementsov-Ogievskiy]

On 14.01.26 15:22, Peter Xu wrote:
> On Wed, Jan 14, 2026 at 09:19:10AM +0300, Vladimir Sementsov-Ogievskiy wrote:
>> On 13.01.26 21:56, Peter Xu wrote:
>>> On Tue, Jan 13, 2026 at 01:12:42PM -0500, Stefan Hajnoczi wrote:
>>>> On Tue, Jan 13, 2026 at 02:58:09PM +0500, Alexandr Moshkov wrote:
>>>>
>>>> Peter: Please review the migration aspects (especially the vmstates).
>>>> Thank you!
>>>
>>> Looks good from my side as long as it's based on VMSD, I appreciate that
>>> change from the old versions where it used to use qemufile APIs.
>>>
>>> The major question here is if this series depends on Vladimir's other
>>> series
>>
>> No, it does not. And if we can proceed with merging these series first, I'll
>> be happy to rebase on top of it.
> 
> I thought it requires migrate_local_vhost_user_blk() be present?  The
> inflight feature should not be enabled only if there's a hint that it's a
> local migration..

Oh right, I missed it.

We discussed that Alexandr will rebase the series on master without dependency
on my RFC.

> 
> I'll comment inline on the patch later.
> 
>>
>>> while there's still one patch that is not-for-merge:
>>>
>>> https://lore.kernel.org/all/20251016114104.1384675-24-vsementsov@yandex-team.ru/#t
>>>
>>> Does it automatically mark this series RFC as well?
>>>
>>> Personally speaking, a new migration cap would work all fine, we should
>>> have discussed it somewhere previously.  Said that, "local-vhost-user-blk"
>>> capability is likely not the right one.  IMHO it should be either "local"
>>> or "fd-passing" / "fd-passthrough" (or something generic) as the name.  If
>>> we are not sure if we will leverage more than "passing the FDs around", we
>>> can make it as simple as "local" as a new migration capability.
>>>
>>> Then migration's misc.h should export a function migrate_is_local() then
>>> device code can probe that in its own vmstate handling paths on save/load.
>>>
>>> A note to Vladimir: please remember to add a check to enforce UNIX socket
>>> when a formal patch 23 will be proposed some day, no matter what is the
>>> name of the capability.  It should fail qmp "migrate" or qmp
>>> "migrate_incoming" command if the main URI is not a unix socket.
>>>
>>
>> Thanks! I'll keep that in mind when prepare next version.
>>
>> -- 
>> Best regards,
>> Vladimir
>>
> 


-- 
Best regards,
Vladimir

Re: [PATCH v6 0/5] support inflight migration

[Peter Xu]

On Wed, Jan 14, 2026 at 05:35:53PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> On 14.01.26 15:22, Peter Xu wrote:
> > On Wed, Jan 14, 2026 at 09:19:10AM +0300, Vladimir Sementsov-Ogievskiy wrote:
> > > On 13.01.26 21:56, Peter Xu wrote:
> > > > On Tue, Jan 13, 2026 at 01:12:42PM -0500, Stefan Hajnoczi wrote:
> > > > > On Tue, Jan 13, 2026 at 02:58:09PM +0500, Alexandr Moshkov wrote:
> > > > > 
> > > > > Peter: Please review the migration aspects (especially the vmstates).
> > > > > Thank you!
> > > > 
> > > > Looks good from my side as long as it's based on VMSD, I appreciate that
> > > > change from the old versions where it used to use qemufile APIs.
> > > > 
> > > > The major question here is if this series depends on Vladimir's other
> > > > series
> > > 
> > > No, it does not. And if we can proceed with merging these series first, I'll
> > > be happy to rebase on top of it.
> > 
> > I thought it requires migrate_local_vhost_user_blk() be present?  The
> > inflight feature should not be enabled only if there's a hint that it's a
> > local migration..
> 
> Oh right, I missed it.
> 
> We discussed that Alexandr will rebase the series on master without dependency
> on my RFC.

The problem is IIUC the new INFLIGHT feature bit will be declared as
supported to vhost-user-block after applying this series.  Then if we start
a remote migration (rather than local) it'll be automatically (and wrongly)
enabled?

AFAIU, the dependency makes sense, at least to the patch to introduce the
"local" / ... capability?

-- 
Peter Xu

Re: [PATCH v6 0/5] support inflight migration

[Vladimir Sementsov-Ogievskiy]

On 14.01.26 18:17, Peter Xu wrote:
> On Wed, Jan 14, 2026 at 05:35:53PM +0300, Vladimir Sementsov-Ogievskiy wrote:
>> On 14.01.26 15:22, Peter Xu wrote:
>>> On Wed, Jan 14, 2026 at 09:19:10AM +0300, Vladimir Sementsov-Ogievskiy wrote:
>>>> On 13.01.26 21:56, Peter Xu wrote:
>>>>> On Tue, Jan 13, 2026 at 01:12:42PM -0500, Stefan Hajnoczi wrote:
>>>>>> On Tue, Jan 13, 2026 at 02:58:09PM +0500, Alexandr Moshkov wrote:
>>>>>>
>>>>>> Peter: Please review the migration aspects (especially the vmstates).
>>>>>> Thank you!
>>>>>
>>>>> Looks good from my side as long as it's based on VMSD, I appreciate that
>>>>> change from the old versions where it used to use qemufile APIs.
>>>>>
>>>>> The major question here is if this series depends on Vladimir's other
>>>>> series
>>>>
>>>> No, it does not. And if we can proceed with merging these series first, I'll
>>>> be happy to rebase on top of it.
>>>
>>> I thought it requires migrate_local_vhost_user_blk() be present?  The
>>> inflight feature should not be enabled only if there's a hint that it's a
>>> local migration..
>>
>> Oh right, I missed it.
>>
>> We discussed that Alexandr will rebase the series on master without dependency
>> on my RFC.
> 
> The problem is IIUC the new INFLIGHT feature bit will be declared as
> supported to vhost-user-block after applying this series.  Then if we start
> a remote migration (rather than local) it'll be automatically (and wrongly)
> enabled?
> 

No, not so.

We develop inflight-region migraiton (this series) exactly for remote migration,
not for local.

My series about backend transfer (fd-migration) will migrate inflight-region the
other way - by migrating its FD.

> AFAIU, the dependency makes sense, at least to the patch to introduce the
> "local" / ... capability?
> 


-- 
Best regards,
Vladimir

Re: [PATCH v6 0/5] support inflight migration

[Peter Xu]

On Wed, Jan 14, 2026 at 08:54:50PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> On 14.01.26 18:17, Peter Xu wrote:
> > On Wed, Jan 14, 2026 at 05:35:53PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> > > On 14.01.26 15:22, Peter Xu wrote:
> > > > On Wed, Jan 14, 2026 at 09:19:10AM +0300, Vladimir Sementsov-Ogievskiy wrote:
> > > > > On 13.01.26 21:56, Peter Xu wrote:
> > > > > > On Tue, Jan 13, 2026 at 01:12:42PM -0500, Stefan Hajnoczi wrote:
> > > > > > > On Tue, Jan 13, 2026 at 02:58:09PM +0500, Alexandr Moshkov wrote:
> > > > > > > 
> > > > > > > Peter: Please review the migration aspects (especially the vmstates).
> > > > > > > Thank you!
> > > > > > 
> > > > > > Looks good from my side as long as it's based on VMSD, I appreciate that
> > > > > > change from the old versions where it used to use qemufile APIs.
> > > > > > 
> > > > > > The major question here is if this series depends on Vladimir's other
> > > > > > series
> > > > > 
> > > > > No, it does not. And if we can proceed with merging these series first, I'll
> > > > > be happy to rebase on top of it.
> > > > 
> > > > I thought it requires migrate_local_vhost_user_blk() be present?  The
> > > > inflight feature should not be enabled only if there's a hint that it's a
> > > > local migration..
> > > 
> > > Oh right, I missed it.
> > > 
> > > We discussed that Alexandr will rebase the series on master without dependency
> > > on my RFC.
> > 
> > The problem is IIUC the new INFLIGHT feature bit will be declared as
> > supported to vhost-user-block after applying this series.  Then if we start
> > a remote migration (rather than local) it'll be automatically (and wrongly)
> > enabled?
> > 
> 
> No, not so.
> 
> We develop inflight-region migraiton (this series) exactly for remote migration,
> not for local.
> 
> My series about backend transfer (fd-migration) will migrate inflight-region the
> other way - by migrating its FD.

Ah, I see.. Yes I think you're right then there's no dependency needed.
Please go ahead with the repost.

Thanks,

-- 
Peter Xu

Re: [PATCH v6 4/5] vhost: add vmstate for inflight region with inner buffer

[Peter Xu]

On Tue, Jan 13, 2026 at 02:58:17PM +0500, Alexandr Moshkov wrote:
> Prepare for future inflight region migration for vhost-user-blk.
> We need to migrate size, queue_size, and inner buffer.
> 
> So firstly it migrate size and queue_size fields, then allocate memory for buffer with
> migrated size, then migrate inner buffer itself.
> 
> Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
> ---
>  hw/virtio/vhost.c         | 57 +++++++++++++++++++++++++++++++++++++++
>  include/hw/virtio/vhost.h |  6 +++++
>  2 files changed, 63 insertions(+)
> 
> diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
> index c46203eb9c..f655c53b67 100644
> --- a/hw/virtio/vhost.c
> +++ b/hw/virtio/vhost.c
> @@ -2028,6 +2028,63 @@ const VMStateDescription vmstate_backend_transfer_vhost_inflight = {
>      }
>  };
>  
> +static int vhost_inflight_buffer_pre_load(void *opaque, Error **errp)
> +{
> +    info_report("vhost_inflight_region_buffer_pre_load");
> +    struct vhost_inflight *inflight = opaque;
> +
> +    int fd = -1;
> +    void *addr = qemu_memfd_alloc("vhost-inflight", inflight->size,
> +                                  F_SEAL_GROW | F_SEAL_SHRINK | F_SEAL_SEAL,
> +                                  &fd, errp);
> +    if (*errp) {
> +        return -ENOMEM;
> +    }
> +
> +    inflight->offset = 0;
> +    inflight->addr = addr;
> +    inflight->fd = fd;
> +
> +    return 0;
> +}
> +
> +const VMStateDescription vmstate_vhost_inflight_region_buffer = {
> +    .name = "vhost-inflight-region/buffer",
> +    .pre_load_errp = vhost_inflight_buffer_pre_load,
> +    .fields = (const VMStateField[]) {
> +        VMSTATE_VBUFFER_UINT64(addr, struct vhost_inflight, 0, NULL, size),
> +        VMSTATE_END_OF_LIST()
> +    }
> +};
> +
> +static int vhost_inflight_region_post_load(void *opaque,
> +                                           int version_id,
> +                                           Error **errp)
> +{
> +    struct vhost_inflight *inflight = opaque;
> +
> +    if (inflight->addr == NULL) {

IIUC this can never happen because pre_load() must trigger before
post_load(), and when reaching post_load() it means pre_load() must have
succeeded..

So, IIUC we can drop this post_load() completely (or assert addr in
pre_load instead).

> +        error_setg(errp, "inflight buffer subsection has not been loaded");
> +        return -EINVAL;
> +    }
> +
> +    return 0;
> +}
> +
> +const VMStateDescription vmstate_vhost_inflight_region = {
> +    .name = "vhost-inflight-region",
> +    .post_load_errp = vhost_inflight_region_post_load,
> +    .fields = (const VMStateField[]) {
> +        VMSTATE_UINT64(size, struct vhost_inflight),
> +        VMSTATE_UINT16(queue_size, struct vhost_inflight),
> +        VMSTATE_END_OF_LIST()
> +    },
> +    .subsections = (const VMStateDescription * const []) {
> +        &vmstate_vhost_inflight_region_buffer,
> +        NULL
> +    }
> +};
> +
>  const VMStateDescription vmstate_vhost_virtqueue = {
>      .name = "vhost-virtqueue",
>      .fields = (const VMStateField[]) {
> diff --git a/include/hw/virtio/vhost.h b/include/hw/virtio/vhost.h
> index 13ca2c319f..dd552de91f 100644
> --- a/include/hw/virtio/vhost.h
> +++ b/include/hw/virtio/vhost.h
> @@ -596,6 +596,12 @@ extern const VMStateDescription vmstate_backend_transfer_vhost_inflight;
>                             vmstate_backend_transfer_vhost_inflight, \
>                             struct vhost_inflight)
>  
> +extern const VMStateDescription vmstate_vhost_inflight_region;
> +#define VMSTATE_VHOST_INFLIGHT_REGION(_field, _state) \
> +    VMSTATE_STRUCT_POINTER(_field, _state, \
> +                           vmstate_vhost_inflight_region, \
> +                           struct vhost_inflight)
> +
>  extern const VMStateDescription vmstate_vhost_dev;
>  #define VMSTATE_BACKEND_TRANSFER_VHOST(_field, _state) \
>      VMSTATE_STRUCT(_field, _state, 0, vmstate_vhost_dev, struct vhost_dev)
> -- 
> 2.34.1
> 

-- 
Peter Xu

Re: [PATCH v6 4/5] vhost: add vmstate for inflight region with inner buffer

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2796 bytes --]

On Wed, Jan 14, 2026 at 02:15:27PM -0500, Peter Xu wrote:
> On Tue, Jan 13, 2026 at 02:58:17PM +0500, Alexandr Moshkov wrote:
> > Prepare for future inflight region migration for vhost-user-blk.
> > We need to migrate size, queue_size, and inner buffer.
> > 
> > So firstly it migrate size and queue_size fields, then allocate memory for buffer with
> > migrated size, then migrate inner buffer itself.
> > 
> > Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
> > ---
> >  hw/virtio/vhost.c         | 57 +++++++++++++++++++++++++++++++++++++++
> >  include/hw/virtio/vhost.h |  6 +++++
> >  2 files changed, 63 insertions(+)
> > 
> > diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
> > index c46203eb9c..f655c53b67 100644
> > --- a/hw/virtio/vhost.c
> > +++ b/hw/virtio/vhost.c
> > @@ -2028,6 +2028,63 @@ const VMStateDescription vmstate_backend_transfer_vhost_inflight = {
> >      }
> >  };
> >  
> > +static int vhost_inflight_buffer_pre_load(void *opaque, Error **errp)
> > +{
> > +    info_report("vhost_inflight_region_buffer_pre_load");
> > +    struct vhost_inflight *inflight = opaque;
> > +
> > +    int fd = -1;
> > +    void *addr = qemu_memfd_alloc("vhost-inflight", inflight->size,
> > +                                  F_SEAL_GROW | F_SEAL_SHRINK | F_SEAL_SEAL,
> > +                                  &fd, errp);
> > +    if (*errp) {
> > +        return -ENOMEM;
> > +    }
> > +
> > +    inflight->offset = 0;
> > +    inflight->addr = addr;
> > +    inflight->fd = fd;
> > +
> > +    return 0;
> > +}
> > +
> > +const VMStateDescription vmstate_vhost_inflight_region_buffer = {
> > +    .name = "vhost-inflight-region/buffer",
> > +    .pre_load_errp = vhost_inflight_buffer_pre_load,
> > +    .fields = (const VMStateField[]) {
> > +        VMSTATE_VBUFFER_UINT64(addr, struct vhost_inflight, 0, NULL, size),
> > +        VMSTATE_END_OF_LIST()
> > +    }
> > +};
> > +
> > +static int vhost_inflight_region_post_load(void *opaque,
> > +                                           int version_id,
> > +                                           Error **errp)
> > +{
> > +    struct vhost_inflight *inflight = opaque;
> > +
> > +    if (inflight->addr == NULL) {
> 
> IIUC this can never happen because pre_load() must trigger before
> post_load(), and when reaching post_load() it means pre_load() must have
> succeeded..
> 
> So, IIUC we can drop this post_load() completely (or assert addr in
> pre_load instead).

I asked for this input validation check. If the migration stream is
inconsistent (e.g. broken or malicious source QEMU), then the subsection
might be missing but size could be non-zero. The destination QEMU should
fail cleanly and not run into undefined behavior.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v6 4/5] vhost: add vmstate for inflight region with inner buffer

[Peter Xu]

On Wed, Jan 14, 2026 at 04:38:17PM -0500, Stefan Hajnoczi wrote:
> On Wed, Jan 14, 2026 at 02:15:27PM -0500, Peter Xu wrote:
> > On Tue, Jan 13, 2026 at 02:58:17PM +0500, Alexandr Moshkov wrote:
> > > Prepare for future inflight region migration for vhost-user-blk.
> > > We need to migrate size, queue_size, and inner buffer.
> > > 
> > > So firstly it migrate size and queue_size fields, then allocate memory for buffer with
> > > migrated size, then migrate inner buffer itself.
> > > 
> > > Signed-off-by: Alexandr Moshkov <dtalexundeer@yandex-team.ru>
> > > ---
> > >  hw/virtio/vhost.c         | 57 +++++++++++++++++++++++++++++++++++++++
> > >  include/hw/virtio/vhost.h |  6 +++++
> > >  2 files changed, 63 insertions(+)
> > > 
> > > diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
> > > index c46203eb9c..f655c53b67 100644
> > > --- a/hw/virtio/vhost.c
> > > +++ b/hw/virtio/vhost.c
> > > @@ -2028,6 +2028,63 @@ const VMStateDescription vmstate_backend_transfer_vhost_inflight = {
> > >      }
> > >  };
> > >  
> > > +static int vhost_inflight_buffer_pre_load(void *opaque, Error **errp)
> > > +{
> > > +    info_report("vhost_inflight_region_buffer_pre_load");
> > > +    struct vhost_inflight *inflight = opaque;
> > > +
> > > +    int fd = -1;
> > > +    void *addr = qemu_memfd_alloc("vhost-inflight", inflight->size,
> > > +                                  F_SEAL_GROW | F_SEAL_SHRINK | F_SEAL_SEAL,
> > > +                                  &fd, errp);
> > > +    if (*errp) {
> > > +        return -ENOMEM;
> > > +    }
> > > +
> > > +    inflight->offset = 0;
> > > +    inflight->addr = addr;
> > > +    inflight->fd = fd;
> > > +
> > > +    return 0;
> > > +}
> > > +
> > > +const VMStateDescription vmstate_vhost_inflight_region_buffer = {
> > > +    .name = "vhost-inflight-region/buffer",
> > > +    .pre_load_errp = vhost_inflight_buffer_pre_load,
> > > +    .fields = (const VMStateField[]) {
> > > +        VMSTATE_VBUFFER_UINT64(addr, struct vhost_inflight, 0, NULL, size),
> > > +        VMSTATE_END_OF_LIST()
> > > +    }
> > > +};
> > > +
> > > +static int vhost_inflight_region_post_load(void *opaque,
> > > +                                           int version_id,
> > > +                                           Error **errp)
> > > +{
> > > +    struct vhost_inflight *inflight = opaque;
> > > +
> > > +    if (inflight->addr == NULL) {
> > 
> > IIUC this can never happen because pre_load() must trigger before
> > post_load(), and when reaching post_load() it means pre_load() must have
> > succeeded..
> > 
> > So, IIUC we can drop this post_load() completely (or assert addr in
> > pre_load instead).
> 
> I asked for this input validation check. If the migration stream is
> inconsistent (e.g. broken or malicious source QEMU), then the subsection
> might be missing but size could be non-zero. The destination QEMU should
> fail cleanly and not run into undefined behavior.

Ah I misread it as the one pairing with the pre_load().  It makes sense
indeed to have such post_load() in the parent VMSD.

Please ignore my comment, sorry for the noise.

-- 
Peter Xu