* [PATCH 1/8] vfio-user: add live migration to vfio-user protocol specification
2026-09-03 13:08 [PATCH 0/8] vfio-user: live migration support Hugo Komatsu
@ 2026-09-03 13:08 ` Hugo Komatsu
2026-09-08 14:41 ` John Levon
2026-09-03 13:08 ` [PATCH 2/8] docs/vfio-user: clarify data_fd and DMA logging behaviors Hugo Komatsu
` (6 subsequent siblings)
7 siblings, 1 reply; 17+ messages in thread
From: Hugo Komatsu @ 2026-09-03 13:08 UTC (permalink / raw)
To: qemu-devel@nongnu.org
Cc: qemu-s390x@nongnu.org, Alex Williamson, Cédric Le Goater,
Cornelia Huck, Eric Farman, Halil Pasic, Jason Herne, John Levon,
Mark Cave-Ayland, Matthew Rosato, Pierrick Bouvier,
Rafael Castillo, Thanos Makatos, Tony Krowiak
From: William Henderson <william.henderson@nutanix.com>
This patch adds live migration to the vfio-user specification, based
on the new VFIO migration interface introduced in the kernel here:
https://lore.kernel.org/all/20220224142024.147653-10-yishaih@nvidia.com/
We differ from the VFIO protocol in that, while VFIO transfers
migration data using a file descriptor, we simply use the already
established vfio-user socket with three additional commands:
VFIO_USER_MIG_DATA_READ, VFIO_USER_MIG_DATA_WRITE, and
VFIO_USER_GET_PRECOPY_INFO, which have stream semantics. We also
don't use P2P states as we don't yet have a use-case for them,
although this may change in the future.
Signed-off-by: William Henderson <william.henderson@nutanix.com>
---
docs/interop/vfio-user.rst | 120 ++++++++++++++++++++++---------------
1 file changed, 73 insertions(+), 47 deletions(-)
diff --git a/docs/interop/vfio-user.rst b/docs/interop/vfio-user.rst
index 1c92f442d2..73282cd2dd 100644
--- a/docs/interop/vfio-user.rst
+++ b/docs/interop/vfio-user.rst
@@ -387,6 +387,7 @@ Name Command Request Direction
``VFIO_USER_DEVICE_FEATURE`` 16 client -> server
``VFIO_USER_MIG_DATA_READ`` 17 client -> server
``VFIO_USER_MIG_DATA_WRITE`` 18 client -> server
+``VFIO_USER_GET_PRECOPY_INFO`` 19 client -> server
====================================== ========= =================
Header
@@ -1588,26 +1589,22 @@ The request payload for this message is a structure of the following format.
+-------+--------+--------------------------------+
* *argsz* is the maximum size of the reply payload.
-
* *flags* defines the action to be performed by the server and upon which
feature:
* The feature index consists of the least significant 16 bits of the flags
field, and can be accessed using the ``VFIO_DEVICE_FEATURE_MASK`` bit mask.
-
* ``VFIO_DEVICE_FEATURE_GET`` instructs the server to get the data for the
given feature.
-
* ``VFIO_DEVICE_FEATURE_SET`` instructs the server to set the feature data to
- that given in the ``data`` field of the payload.
-
+ that given in the data field of the payload.
* ``VFIO_DEVICE_FEATURE_PROBE`` instructs the server to probe for feature
support. If ``VFIO_DEVICE_FEATURE_GET`` and/or ``VFIO_DEVICE_FEATURE_SET``
are also set, the probe will only return success if all of the indicated
methods are supported.
- ``VFIO_DEVICE_FEATURE_GET`` and ``VFIO_DEVICE_FEATURE_SET`` are mutually
- exclusive, except for use with ``VFIO_DEVICE_FEATURE_PROBE``.
+``VFIO_DEVICE_FEATURE_GET`` and ``VFIO_DEVICE_FEATURE_SET`` are mutually
+exclusive, except for use with ``VFIO_DEVICE_FEATURE_PROBE``.
* *data* is specific to the particular feature. It is not used for probing.
@@ -1729,25 +1726,19 @@ server, a subset of those defined in ``<linux/vfio.h>``
* ``RUNNING -> STOP``, ``STOP_COPY -> STOP``: Stop the operation of the device.
The ``STOP_COPY`` arc terminates the data transfer session.
-
* ``RESUMING -> STOP``: Terminate the data transfer session. Complete processing
of the migration data. Stop the operation of the device. If the delivered data
is found to be incomplete, inconsistent, or otherwise invalid, fail the
``SET`` command and optionally transition to the ``ERROR`` state.
-
* ``PRE_COPY -> RUNNING``: Terminate the data transfer session. The device is
now fully operational.
-
* ``STOP -> RUNNING``: Start the operation of the device.
-
* ``RUNNING -> PRE_COPY``, ``STOP -> STOP_COPY``: Begin the process of saving
the device state. The device operation is unchanged, but data transfer begins.
``PRE_COPY`` and ``STOP_COPY`` are referred to as the "saving group" of
states.
-
* ``PRE_COPY -> STOP_COPY``: Continue to transfer migration data, but stop
device operation.
-
* ``STOP -> RESUMING``: Start the process of restoring the device state. The
internal device state may be changed to prepare the device to receive the
migration data.
@@ -1771,7 +1762,6 @@ above FSM arcs. As there are multiple paths, the path should be selected based
on the following rules:
* Select the shortest path.
-
* The path cannot have saving group states as interior arcs, only start/end
states.
@@ -1803,10 +1793,8 @@ The data field of the ``SET`` request is structured as follows:
If the device cannot do the hinted page size then it's the driver's choice
which page size to pick based on its support. On output the device will return
the page size it selected.
-
* *num_ranges* is the number of IOVA ranges to monitor. A value of zero
indicates that all writes should be logged.
-
* *ranges* is an array of ``vfio_user_device_feature_dma_logging_range``
entries:
@@ -1818,8 +1806,8 @@ The data field of the ``SET`` request is structured as follows:
| length | 8 | 8 |
+--------+--------+------+
- * *iova* is the base IO virtual address
- * *length* is the length of the range to log
+* *iova* is the base IO virtual address
+* *length* is the length of the range to log
Upon success, the response data field will be the same as the request, unless
the page size was changed, in which case this will be reflected in the response.
@@ -1844,14 +1832,12 @@ The data field of the request is structured as follows:
+-----------+--------+------+
* *iova* is the base IO virtual address
-
* *length* is the length of the range
-
* *page_size* is the unit of granularity of the bitmap, and must be a power of
two. It doesn't have to match the value given to
``VFIO_DEVICE_FEATURE_DMA_LOGGING_START`` because the driver will format its
internal logging to match the reporting page size possibly by replicating bits
- if the internal page size is lower than requested
+ if the internal page size is lower than requested.
The data field of the response is identical, except with the bitmap added on
the end at offset 24.
@@ -1861,21 +1847,16 @@ reporting a *page_size* unit of IOVA. The bits outside of the requested range
must be zero.
The mapping of IOVA to bits is given by:
-
``bitmap[(addr - iova)/page_size] & (1ULL << (addr % 64))``
``VFIO_USER_MIG_DATA_READ``
---------------------------
-This command is used to read data from the source migration server while it is
-in a saving group state (``PRE_COPY`` or ``STOP_COPY``).
+This command is used to read data from the device while it is in a
+saving-related migration state.
-This command, and ``VFIO_USER_MIG_DATA_WRITE``, are used in place of the
-``data_fd`` file descriptor in ``<linux/vfio.h>``
-(``struct vfio_device_feature_mig_state``) to enable all data transport to use
-the single already-established UNIX socket. Hence, the migration data is
-treated like a stream, so the client must continue reading until no more
-migration data remains.
+This command is analogous to reading from the ``VFIO_MIG_GET_PRECOPY_INFO``
+data_fd.
Request
^^^^^^^
@@ -1890,9 +1871,8 @@ The request payload for this message is a structure of the following format.
| size | 4 | 4 |
+-------+--------+------+
-* *argsz* is the maximum size of the reply payload.
-
-* *size* is the size of the migration data to read.
+* *argsz* is the size of the above structure.
+* *size* is the amount of data to read.
Reply
^^^^^
@@ -1909,21 +1889,18 @@ The reply payload for this message is a structure of the following format.
| data | 8 | variable |
+-------+--------+----------+
-* *argsz* is the size of the above structure, including the size of the data.
-
-* *size* indicates the size of returned migration data. If this is less than the
- requested size, there is no more migration data to read.
-
-* *data* contains the migration data.
+* *argsz* is the size of the above structure.
+* *size* is the amount of data read.
+* *data* is the data read.
``VFIO_USER_MIG_DATA_WRITE``
----------------------------
-This command is used to write data to the destination migration server while it
-is in the ``RESUMING`` state.
+This command is used to write data to the device while it is in a
+resuming-related migration state.
-As above, this replaces the ``data_fd`` file descriptor for transport of
-migration data, and as such, the migration data is treated like a stream.
+This command is analogous to writing to the ``VFIO_MIG_GET_PRECOPY_INFO``
+data_fd.
Request
^^^^^^^
@@ -1940,16 +1917,65 @@ The request payload for this message is a structure of the following format.
| data | 8 | variable |
+-------+--------+----------+
-* *argsz* is the maximum size of the reply payload.
+* *argsz* is the size of the above structure.
+* *size* is the amount of data to write.
+* *data* is the data to write.
+
+Reply
+^^^^^
-* *size* is the size of the migration data to be written.
+The reply payload for this message is a structure of the following format.
-* *data* contains the migration data.
++-------+--------+----------+
+| Name | Offset | Size |
++=======+========+==========+
+| argsz | 0 | 4 |
++-------+--------+----------+
+| size | 4 | 4 |
++-------+--------+----------+
+
+* *argsz* is the size of the above structure.
+* *size* is the amount of data written.
+
+``VFIO_USER_GET_PRECOPY_INFO``
+------------------------------
+
+This command is used to query the migration data transfer progress during the
+``PRE_COPY`` state.
+
+This command is analogous to the ``VFIO_MIG_GET_PRECOPY_INFO`` ioctl.
+
+Request
+^^^^^^^
+
+The request payload for this message is a structure of the following format.
+
++---------------+--------+------+
+| Name | Offset | Size |
++===============+========+======+
+| argsz | 0 | 4 |
++---------------+--------+------+
+| flags | 4 | 4 |
++---------------+--------+------+
+| initial_bytes | 8 | 8 |
++---------------+--------+------+
+| dirty_bytes | 16 | 8 |
++---------------+--------+------+
+
+* *argsz* is the size of the above structure.
+* *flags* is unused.
+* *initial_bytes* is unused in the request.
+* *dirty_bytes* is unused in the request.
Reply
^^^^^
-There is no reply payload for this message.
+The reply payload has the exact same structure as the request payload.
+
+* *argsz* is the size of the above structure.
+* *flags* is unused.
+* *initial_bytes* contains the amount of initial data available for transfer.
+* *dirty_bytes* contains the amount of dirty data available for transfer.
Appendices
==========
--
2.39.3
^ permalink raw reply related [flat|nested] 17+ messages in thread* Re: [PATCH 1/8] vfio-user: add live migration to vfio-user protocol specification
2026-09-03 13:08 ` [PATCH 1/8] vfio-user: add live migration to vfio-user protocol specification Hugo Komatsu
@ 2026-09-08 14:41 ` John Levon
0 siblings, 0 replies; 17+ messages in thread
From: John Levon @ 2026-09-08 14:41 UTC (permalink / raw)
To: Hugo Komatsu
Cc: qemu-devel@nongnu.org, qemu-s390x@nongnu.org, Alex Williamson,
Cédric Le Goater, Cornelia Huck, Eric Farman, Halil Pasic,
Jason Herne, Mark Cave-Ayland, Matthew Rosato, Pierrick Bouvier,
Rafael Castillo, Thanos Makatos, Tony Krowiak
On Thu, Sep 03, 2026 at 02:08:40PM +0100, Hugo Komatsu wrote:
> From: William Henderson <william.henderson@nutanix.com>
>
> This patch adds live migration to the vfio-user specification, based
> on the new VFIO migration interface introduced in the kernel here:
> https://lore.kernel.org/all/20220224142024.147653-10-yishaih@nvidia.com/
>
> We differ from the VFIO protocol in that, while VFIO transfers
> migration data using a file descriptor, we simply use the already
> established vfio-user socket with three additional commands:
> VFIO_USER_MIG_DATA_READ, VFIO_USER_MIG_DATA_WRITE, and
> VFIO_USER_GET_PRECOPY_INFO, which have stream semantics. We also
> don't use P2P states as we don't yet have a use-case for them,
> although this may change in the future.
>
> Signed-off-by: William Henderson <william.henderson@nutanix.com>
Reviewed-by: John Levon <john.levon@nutanix.com>
^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH 2/8] docs/vfio-user: clarify data_fd and DMA logging behaviors
2026-09-03 13:08 [PATCH 0/8] vfio-user: live migration support Hugo Komatsu
2026-09-03 13:08 ` [PATCH 1/8] vfio-user: add live migration to vfio-user protocol specification Hugo Komatsu
@ 2026-09-03 13:08 ` Hugo Komatsu
2026-09-08 14:42 ` John Levon
2026-09-03 13:08 ` [PATCH 3/8] vfio: route device_feature through io_ops Hugo Komatsu
` (5 subsequent siblings)
7 siblings, 1 reply; 17+ messages in thread
From: Hugo Komatsu @ 2026-09-03 13:08 UTC (permalink / raw)
To: qemu-devel@nongnu.org
Cc: qemu-s390x@nongnu.org, Alex Williamson, Cédric Le Goater,
Cornelia Huck, Eric Farman, Halil Pasic, Jason Herne, John Levon,
Mark Cave-Ayland, Matthew Rosato, Pierrick Bouvier,
Rafael Castillo, Thanos Makatos, Tony Krowiak
Specify that the data_fd field must be explicitly set to -1 by the
server, rather than relying on client-side fixups.
Clarify that the server must return an empty bitmap rather than
an error when reporting dirty pages for unmapped or message-based
DMA regions, as the client tracks these natively.
Signed-off-by: Hugo Komatsu <hugo.komatsu@nutanix.com>
---
docs/interop/vfio-user.rst | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)
diff --git a/docs/interop/vfio-user.rst b/docs/interop/vfio-user.rst
index 73282cd2dd..17c0f82355 100644
--- a/docs/interop/vfio-user.rst
+++ b/docs/interop/vfio-user.rst
@@ -1712,9 +1712,10 @@ structured as follows:
| VFIO_DEVICE_STATE_PRE_COPY_P2P | 7 | (not used in vfio-user) |
+--------------------------------+-------+---------------------------------------------------------------------+
-* *data_fd* is unused in vfio-user, as the ``VFIO_USER_MIG_DATA_READ`` and
- ``VFIO_USER_MIG_DATA_WRITE`` messages are used instead for migration data
- transport.
+* *data_fd* must be set to -1 in both the request and the response. It
+ is unused in vfio-user, as the ``VFIO_USER_MIG_DATA_READ`` and
+ ``VFIO_USER_MIG_DATA_WRITE`` messages are used instead for migration
+ data transport.
Direct State Transitions
""""""""""""""""""""""""
@@ -1846,6 +1847,11 @@ The bitmap is an array of u64s that holds the output bitmap, with 1 bit
reporting a *page_size* unit of IOVA. The bits outside of the requested range
must be zero.
+If the server cannot translate the requested IOVA range (e.g. because it
+is unmapped, or accessed via message-based DMA), the server must return
+an empty (zeroed) bitmap rather than returning an error. The client
+natively tracks dirty pages for message-based DMA operations.
+
The mapping of IOVA to bits is given by:
``bitmap[(addr - iova)/page_size] & (1ULL << (addr % 64))``
--
2.39.3
^ permalink raw reply related [flat|nested] 17+ messages in thread* Re: [PATCH 2/8] docs/vfio-user: clarify data_fd and DMA logging behaviors
2026-09-03 13:08 ` [PATCH 2/8] docs/vfio-user: clarify data_fd and DMA logging behaviors Hugo Komatsu
@ 2026-09-08 14:42 ` John Levon
0 siblings, 0 replies; 17+ messages in thread
From: John Levon @ 2026-09-08 14:42 UTC (permalink / raw)
To: Hugo Komatsu
Cc: qemu-devel@nongnu.org, qemu-s390x@nongnu.org, Alex Williamson,
Cédric Le Goater, Cornelia Huck, Eric Farman, Halil Pasic,
Jason Herne, Mark Cave-Ayland, Matthew Rosato, Pierrick Bouvier,
Rafael Castillo, Thanos Makatos, Tony Krowiak
On Thu, Sep 03, 2026 at 02:08:42PM +0100, Hugo Komatsu wrote:
> Specify that the data_fd field must be explicitly set to -1 by the
> server, rather than relying on client-side fixups.
>
> Clarify that the server must return an empty bitmap rather than
> an error when reporting dirty pages for unmapped or message-based
> DMA regions, as the client tracks these natively.
>
> Signed-off-by: Hugo Komatsu <hugo.komatsu@nutanix.com>
Reviewed-by: John Levon <john.levon@nutanix.com>
^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH 3/8] vfio: route device_feature through io_ops
2026-09-03 13:08 [PATCH 0/8] vfio-user: live migration support Hugo Komatsu
2026-09-03 13:08 ` [PATCH 1/8] vfio-user: add live migration to vfio-user protocol specification Hugo Komatsu
2026-09-03 13:08 ` [PATCH 2/8] docs/vfio-user: clarify data_fd and DMA logging behaviors Hugo Komatsu
@ 2026-09-03 13:08 ` Hugo Komatsu
2026-09-03 15:35 ` Cédric Le Goater
2026-09-03 13:08 ` [PATCH 4/8] vfio: route device_reset " Hugo Komatsu
` (4 subsequent siblings)
7 siblings, 1 reply; 17+ messages in thread
From: Hugo Komatsu @ 2026-09-03 13:08 UTC (permalink / raw)
To: qemu-devel@nongnu.org
Cc: qemu-s390x@nongnu.org, Alex Williamson, Cédric Le Goater,
Cornelia Huck, Eric Farman, Halil Pasic, Jason Herne, John Levon,
Mark Cave-Ayland, Matthew Rosato, Pierrick Bouvier,
Rafael Castillo, Thanos Makatos, Tony Krowiak
The generic VFIO migration code in hw/vfio/migration.c issue kernel
ioctls directly for VFIO_DEVICE_FEATURE. Route these through the
existing VFIODeviceIOOps table so an alternate transport (vfio-user)
can supply its own implementation.
Implement the kernel version as a thin wrapper around the existing
ioctl() call. This is a pure preparatory refactor: the kernel path
is unchanged.
Co-authored-by: Rafael Castillo <rafael@trfs.me>
Signed-off-by: Hugo Komatsu <hugo.komatsu@nutanix.com>
---
hw/vfio/migration.c | 29 ++++++++++++++---------------
1 file changed, 14 insertions(+), 15 deletions(-)
diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c
index 73c49d8c24..ea6734823b 100644
--- a/hw/vfio/migration.c
+++ b/hw/vfio/migration.c
@@ -168,33 +168,30 @@ int vfio_migration_set_state(VFIODevice *vbasedev,
feature->flags =
VFIO_DEVICE_FEATURE_SET | VFIO_DEVICE_FEATURE_MIG_DEVICE_STATE;
mig_state->device_state = new_state;
- if (ioctl(vbasedev->fd, VFIO_DEVICE_FEATURE, feature)) {
+ ret = vbasedev->io_ops->device_feature(vbasedev, feature);
+ if (ret) {
/* Try to set the device in some good state */
- ret = -errno;
-
if (recover_state == VFIO_DEVICE_STATE_ERROR) {
- error_setg_errno(errp, errno,
+ error_setg_errno(errp, -ret,
"%s Recover state is ERROR. Resetting device",
error_prefix);
-
goto reset_device;
}
- error_setg_errno(errp, errno,
+ error_setg_errno(errp, -ret,
"%s Setting device in recover state %s",
error_prefix, mig_state_to_str(recover_state));
mig_state->device_state = recover_state;
- if (ioctl(vbasedev->fd, VFIO_DEVICE_FEATURE, feature)) {
- ret = -errno;
+ int recover_ret = vbasedev->io_ops->device_feature(vbasedev, feature);
+ if (recover_ret) {
/*
* If setting the device in recover state fails, report
* the error here and propagate the first error.
*/
error_report(
"%s: Failed setting device in recover state, err: %s. Resetting device",
- vbasedev->name, strerror(errno));
-
+ vbasedev->name, strerror(-recover_ret));
goto reset_device;
}
@@ -347,13 +344,13 @@ static int vfio_query_stop_copy_size(VFIODevice *vbasedev)
feature->flags =
VFIO_DEVICE_FEATURE_GET | VFIO_DEVICE_FEATURE_MIG_DATA_SIZE;
- if (ioctl(vbasedev->fd, VFIO_DEVICE_FEATURE, feature)) {
+ ret = vbasedev->io_ops->device_feature(vbasedev, feature);
+ if (ret) {
/*
* If getting pending migration size fails, VFIO_MIG_STOP_COPY_SIZE
* is reported so downtime limit won't be violated.
*/
migration->stopcopy_size = VFIO_MIG_STOP_COPY_SIZE;
- ret = -errno;
warn_report_once("VFIO device %s ioctl(VFIO_DEVICE_FEATURE) on "
"VFIO_DEVICE_FEATURE_MIG_DATA_SIZE failed (%d)",
vbasedev->name, ret);
@@ -1094,11 +1091,13 @@ static int vfio_migration_query_flags(VFIODevice *vbasedev, uint64_t *mig_flags)
struct vfio_device_feature *feature = (struct vfio_device_feature *)buf;
struct vfio_device_feature_migration *mig =
(struct vfio_device_feature_migration *)feature->data;
+ int ret;
feature->argsz = sizeof(buf);
feature->flags = VFIO_DEVICE_FEATURE_GET | VFIO_DEVICE_FEATURE_MIGRATION;
- if (ioctl(vbasedev->fd, VFIO_DEVICE_FEATURE, feature)) {
- return -errno;
+ ret = vbasedev->io_ops->device_feature(vbasedev, feature);
+ if (ret) {
+ return ret;
}
*mig_flags = mig->flags;
@@ -1137,7 +1136,7 @@ static bool vfio_dma_logging_supported(VFIODevice *vbasedev)
feature->flags = VFIO_DEVICE_FEATURE_PROBE |
VFIO_DEVICE_FEATURE_DMA_LOGGING_START;
- return !ioctl(vbasedev->fd, VFIO_DEVICE_FEATURE, feature);
+ return !vbasedev->io_ops->device_feature(vbasedev, feature);
}
static bool vfio_migration_init(VFIODevice *vbasedev, Error **errp)
--
2.39.3
^ permalink raw reply related [flat|nested] 17+ messages in thread* Re: [PATCH 3/8] vfio: route device_feature through io_ops
2026-09-03 13:08 ` [PATCH 3/8] vfio: route device_feature through io_ops Hugo Komatsu
@ 2026-09-03 15:35 ` Cédric Le Goater
0 siblings, 0 replies; 17+ messages in thread
From: Cédric Le Goater @ 2026-09-03 15:35 UTC (permalink / raw)
To: Hugo Komatsu, qemu-devel@nongnu.org
Cc: qemu-s390x@nongnu.org, Alex Williamson, Cornelia Huck,
Eric Farman, Halil Pasic, Jason Herne, John Levon,
Mark Cave-Ayland, Matthew Rosato, Pierrick Bouvier,
Rafael Castillo, Thanos Makatos, Tony Krowiak
On 9/3/26 15:08, Hugo Komatsu wrote:
> The generic VFIO migration code in hw/vfio/migration.c issue kernel
> ioctls directly for VFIO_DEVICE_FEATURE. Route these through the
> existing VFIODeviceIOOps table so an alternate transport (vfio-user)
> can supply its own implementation.
>
> Implement the kernel version as a thin wrapper around the existing
> ioctl() call. This is a pure preparatory refactor: the kernel path
> is unchanged.
>
> Co-authored-by: Rafael Castillo <rafael@trfs.me>
> Signed-off-by: Hugo Komatsu <hugo.komatsu@nutanix.com>
> ---
> hw/vfio/migration.c | 29 ++++++++++++++---------------
> 1 file changed, 14 insertions(+), 15 deletions(-)
>
> diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c
> index 73c49d8c24..ea6734823b 100644
> --- a/hw/vfio/migration.c
> +++ b/hw/vfio/migration.c
> @@ -168,33 +168,30 @@ int vfio_migration_set_state(VFIODevice *vbasedev,
> feature->flags =
> VFIO_DEVICE_FEATURE_SET | VFIO_DEVICE_FEATURE_MIG_DEVICE_STATE;
> mig_state->device_state = new_state;
> - if (ioctl(vbasedev->fd, VFIO_DEVICE_FEATURE, feature)) {
> + ret = vbasedev->io_ops->device_feature(vbasedev, feature);
please use vfio_device_get_feature()
Thanks,
C.
> + if (ret) {
> /* Try to set the device in some good state */
> - ret = -errno;
> -
> if (recover_state == VFIO_DEVICE_STATE_ERROR) {
> - error_setg_errno(errp, errno,
> + error_setg_errno(errp, -ret,
> "%s Recover state is ERROR. Resetting device",
> error_prefix);
> -
> goto reset_device;
> }
>
> - error_setg_errno(errp, errno,
> + error_setg_errno(errp, -ret,
> "%s Setting device in recover state %s",
> error_prefix, mig_state_to_str(recover_state));
>
> mig_state->device_state = recover_state;
> - if (ioctl(vbasedev->fd, VFIO_DEVICE_FEATURE, feature)) {
> - ret = -errno;
> + int recover_ret = vbasedev->io_ops->device_feature(vbasedev, feature);
> + if (recover_ret) {
> /*
> * If setting the device in recover state fails, report
> * the error here and propagate the first error.
> */
> error_report(
> "%s: Failed setting device in recover state, err: %s. Resetting device",
> - vbasedev->name, strerror(errno));
> -
> + vbasedev->name, strerror(-recover_ret));
> goto reset_device;
> }
>
> @@ -347,13 +344,13 @@ static int vfio_query_stop_copy_size(VFIODevice *vbasedev)
> feature->flags =
> VFIO_DEVICE_FEATURE_GET | VFIO_DEVICE_FEATURE_MIG_DATA_SIZE;
>
> - if (ioctl(vbasedev->fd, VFIO_DEVICE_FEATURE, feature)) {
> + ret = vbasedev->io_ops->device_feature(vbasedev, feature);
> + if (ret) {
> /*
> * If getting pending migration size fails, VFIO_MIG_STOP_COPY_SIZE
> * is reported so downtime limit won't be violated.
> */
> migration->stopcopy_size = VFIO_MIG_STOP_COPY_SIZE;
> - ret = -errno;
> warn_report_once("VFIO device %s ioctl(VFIO_DEVICE_FEATURE) on "
> "VFIO_DEVICE_FEATURE_MIG_DATA_SIZE failed (%d)",
> vbasedev->name, ret);
> @@ -1094,11 +1091,13 @@ static int vfio_migration_query_flags(VFIODevice *vbasedev, uint64_t *mig_flags)
> struct vfio_device_feature *feature = (struct vfio_device_feature *)buf;
> struct vfio_device_feature_migration *mig =
> (struct vfio_device_feature_migration *)feature->data;
> + int ret;
>
> feature->argsz = sizeof(buf);
> feature->flags = VFIO_DEVICE_FEATURE_GET | VFIO_DEVICE_FEATURE_MIGRATION;
> - if (ioctl(vbasedev->fd, VFIO_DEVICE_FEATURE, feature)) {
> - return -errno;
> + ret = vbasedev->io_ops->device_feature(vbasedev, feature);
> + if (ret) {
> + return ret;
> }
>
> *mig_flags = mig->flags;
> @@ -1137,7 +1136,7 @@ static bool vfio_dma_logging_supported(VFIODevice *vbasedev)
> feature->flags = VFIO_DEVICE_FEATURE_PROBE |
> VFIO_DEVICE_FEATURE_DMA_LOGGING_START;
>
> - return !ioctl(vbasedev->fd, VFIO_DEVICE_FEATURE, feature);
> + return !vbasedev->io_ops->device_feature(vbasedev, feature);
> }
>
> static bool vfio_migration_init(VFIODevice *vbasedev, Error **errp)
^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH 4/8] vfio: route device_reset through io_ops
2026-09-03 13:08 [PATCH 0/8] vfio-user: live migration support Hugo Komatsu
` (2 preceding siblings ...)
2026-09-03 13:08 ` [PATCH 3/8] vfio: route device_feature through io_ops Hugo Komatsu
@ 2026-09-03 13:08 ` Hugo Komatsu
2026-09-03 15:44 ` Cédric Le Goater
2026-09-03 13:08 ` [PATCH 5/8] vfio: use io_ops abstraction for migration related ioctls Hugo Komatsu
` (3 subsequent siblings)
7 siblings, 1 reply; 17+ messages in thread
From: Hugo Komatsu @ 2026-09-03 13:08 UTC (permalink / raw)
To: qemu-devel@nongnu.org
Cc: qemu-s390x@nongnu.org, Alex Williamson, Cédric Le Goater,
Cornelia Huck, Eric Farman, Halil Pasic, Jason Herne, John Levon,
Mark Cave-Ayland, Matthew Rosato, Pierrick Bouvier,
Rafael Castillo, Thanos Makatos, Tony Krowiak
Various subsystems across QEMU issue kernel ioctls directly for
VFIO_DEVICE_RESET. Route these through a new device_reset slot in the
VFIODeviceIOOps table so an alternate transport (vfio-user) can supply
its own implementation.
Implement the kernel version as a thin wrapper around the existing
ioctl() call. This is a pure preparatory refactor: the kernel path
is unchanged.
Co-authored-by: Rafael Castillo <rafael@trfs.me>
Signed-off-by: Hugo Komatsu <hugo.komatsu@nutanix.com>
---
hw/vfio/ap.c | 4 ++--
hw/vfio/ccw.c | 2 +-
hw/vfio/device.c | 13 +++++++++++--
hw/vfio/migration.c | 7 ++++---
hw/vfio/pci.c | 4 ++--
include/hw/vfio/vfio-device.h | 11 +++++++++++
6 files changed, 31 insertions(+), 10 deletions(-)
diff --git a/hw/vfio/ap.c b/hw/vfio/ap.c
index 6e2a1223ea..3f7652b4e2 100644
--- a/hw/vfio/ap.c
+++ b/hw/vfio/ap.c
@@ -288,10 +288,10 @@ static void vfio_ap_reset(DeviceState *dev)
int ret;
VFIOAPDevice *vapdev = VFIO_AP_DEVICE(dev);
- ret = ioctl(vapdev->vdev.fd, VFIO_DEVICE_RESET);
+ ret = vapdev->vdev.io_ops->device_reset(&vapdev->vdev);
if (ret) {
error_report("%s: failed to reset %s device: %s", __func__,
- vapdev->vdev.name, strerror(errno));
+ vapdev->vdev.name, strerror(-ret));
}
}
diff --git a/hw/vfio/ccw.c b/hw/vfio/ccw.c
index c3dc7c1962..6d5307e568 100644
--- a/hw/vfio/ccw.c
+++ b/hw/vfio/ccw.c
@@ -241,7 +241,7 @@ static void vfio_ccw_reset(DeviceState *dev)
{
VFIOCCWDevice *vcdev = VFIO_CCW(dev);
- ioctl(vcdev->vdev.fd, VFIO_DEVICE_RESET);
+ vcdev->vdev.io_ops->device_reset(&vcdev->vdev);
}
static void vfio_ccw_crw_read(VFIOCCWDevice *vcdev)
diff --git a/hw/vfio/device.c b/hw/vfio/device.c
index 4f11959633..e7d293710e 100644
--- a/hw/vfio/device.c
+++ b/hw/vfio/device.c
@@ -578,9 +578,18 @@ int vfio_device_get_feature(VFIODevice *vbasedev,
}
/*
- * Traditional ioctl() based io
+ * VFIODeviceIOOps implementation for the kernel VFIO backend.
*/
+static int vfio_device_io_device_reset(VFIODevice *vbasedev)
+{
+ int ret;
+
+ ret = ioctl(vbasedev->fd, VFIO_DEVICE_RESET);
+
+ return ret < 0 ? -errno : ret;
+}
+
static int vfio_device_io_device_feature(VFIODevice *vbasedev,
struct vfio_device_feature *feature)
{
@@ -659,7 +668,7 @@ static int vfio_device_io_region_write(VFIODevice *vbasedev, uint8_t index,
static VFIODeviceIOOps vfio_device_io_ops_ioctl = {
.capabilities = VFIO_IO_CAP_DMA_BUF,
-
+ .device_reset = vfio_device_io_device_reset,
.device_feature = vfio_device_io_device_feature,
.get_region_info = vfio_device_io_get_region_info,
.get_irq_info = vfio_device_io_get_irq_info,
diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c
index ea6734823b..1941678e29 100644
--- a/hw/vfio/migration.c
+++ b/hw/vfio/migration.c
@@ -142,7 +142,7 @@ int vfio_migration_set_state(VFIODevice *vbasedev,
struct vfio_device_feature *feature = (struct vfio_device_feature *)buf;
struct vfio_device_feature_mig_state *mig_state =
(struct vfio_device_feature_mig_state *)feature->data;
- int ret;
+ int ret, reset_ret;
g_autofree char *error_prefix =
g_strdup_printf("%s: Failed setting device state to %s.",
vbasedev->name, mig_state_to_str(new_state));
@@ -219,9 +219,10 @@ int vfio_migration_set_state(VFIODevice *vbasedev,
return 0;
reset_device:
- if (ioctl(vbasedev->fd, VFIO_DEVICE_RESET)) {
+ reset_ret = vbasedev->io_ops->device_reset(vbasedev);
+ if (reset_ret) {
hw_error("%s: Failed resetting device, err: %s", vbasedev->name,
- strerror(errno));
+ strerror(-reset_ret));
}
vfio_migration_set_device_state(vbasedev, VFIO_DEVICE_STATE_RUNNING);
diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
index 428ab2f069..12f2974d19 100644
--- a/hw/vfio/pci.c
+++ b/hw/vfio/pci.c
@@ -3785,7 +3785,7 @@ static void vfio_pci_reset(DeviceState *dev)
if (vdev->vbasedev.reset_works &&
(vdev->has_flr || !vdev->has_pm_reset) &&
- !ioctl(vdev->vbasedev.fd, VFIO_DEVICE_RESET)) {
+ !vdev->vbasedev.io_ops->device_reset(&vdev->vbasedev)) {
trace_vfio_pci_reset_flr(vdev->vbasedev.name);
goto post_reset;
}
@@ -3797,7 +3797,7 @@ static void vfio_pci_reset(DeviceState *dev)
/* If nothing else works and the device supports PM reset, use it */
if (vdev->vbasedev.reset_works && vdev->has_pm_reset &&
- !ioctl(vdev->vbasedev.fd, VFIO_DEVICE_RESET)) {
+ !vdev->vbasedev.io_ops->device_reset(&vdev->vbasedev)) {
trace_vfio_pci_reset_pm(vdev->vbasedev.name);
goto post_reset;
}
diff --git a/include/hw/vfio/vfio-device.h b/include/hw/vfio/vfio-device.h
index 8472420d3f..a8fed550eb 100644
--- a/include/hw/vfio/vfio-device.h
+++ b/include/hw/vfio/vfio-device.h
@@ -191,6 +191,17 @@ struct VFIODeviceIOOps {
*/
uint64_t capabilities;
+ /**
+ * @device_reset
+ *
+ * Perform a device reset.
+ *
+ * @vdev: #VFIODevice to use
+ *
+ * Returns 0 on success or -errno.
+ */
+ int (*device_reset)(VFIODevice *vdev);
+
/**
* @device_feature
*
--
2.39.3
^ permalink raw reply related [flat|nested] 17+ messages in thread* Re: [PATCH 4/8] vfio: route device_reset through io_ops
2026-09-03 13:08 ` [PATCH 4/8] vfio: route device_reset " Hugo Komatsu
@ 2026-09-03 15:44 ` Cédric Le Goater
0 siblings, 0 replies; 17+ messages in thread
From: Cédric Le Goater @ 2026-09-03 15:44 UTC (permalink / raw)
To: Hugo Komatsu, qemu-devel@nongnu.org
Cc: qemu-s390x@nongnu.org, Alex Williamson, Cornelia Huck,
Eric Farman, Halil Pasic, Jason Herne, John Levon,
Mark Cave-Ayland, Matthew Rosato, Pierrick Bouvier,
Rafael Castillo, Thanos Makatos, Tony Krowiak
On 9/3/26 15:08, Hugo Komatsu wrote:
> Various subsystems across QEMU issue kernel ioctls directly for
> VFIO_DEVICE_RESET. Route these through a new device_reset slot in the
> VFIODeviceIOOps table so an alternate transport (vfio-user) can supply
> its own implementation.
>
> Implement the kernel version as a thin wrapper around the existing
> ioctl() call. This is a pure preparatory refactor: the kernel path
> is unchanged.
>
> Co-authored-by: Rafael Castillo <rafael@trfs.me>
> Signed-off-by: Hugo Komatsu <hugo.komatsu@nutanix.com>
This should introduce a vfio_device_reset() wrapper (with the null
guard) just like vfio_device_get_feature(), and all callers should
use it.
> ---
> hw/vfio/ap.c | 4 ++--
> hw/vfio/ccw.c | 2 +-
> hw/vfio/device.c | 13 +++++++++++--
> hw/vfio/migration.c | 7 ++++---
> hw/vfio/pci.c | 4 ++--
> include/hw/vfio/vfio-device.h | 11 +++++++++++
> 6 files changed, 31 insertions(+), 10 deletions(-)
>
> diff --git a/hw/vfio/ap.c b/hw/vfio/ap.c
> index 6e2a1223ea..3f7652b4e2 100644
> --- a/hw/vfio/ap.c
> +++ b/hw/vfio/ap.c
> @@ -288,10 +288,10 @@ static void vfio_ap_reset(DeviceState *dev)
> int ret;
> VFIOAPDevice *vapdev = VFIO_AP_DEVICE(dev);
>
> - ret = ioctl(vapdev->vdev.fd, VFIO_DEVICE_RESET);
> + ret = vapdev->vdev.io_ops->device_reset(&vapdev->vdev);
> if (ret) {
> error_report("%s: failed to reset %s device: %s", __func__,
> - vapdev->vdev.name, strerror(errno));
> + vapdev->vdev.name, strerror(-ret));
> }
> }
>
> diff --git a/hw/vfio/ccw.c b/hw/vfio/ccw.c
> index c3dc7c1962..6d5307e568 100644
> --- a/hw/vfio/ccw.c
> +++ b/hw/vfio/ccw.c
> @@ -241,7 +241,7 @@ static void vfio_ccw_reset(DeviceState *dev)
> {
> VFIOCCWDevice *vcdev = VFIO_CCW(dev);
>
> - ioctl(vcdev->vdev.fd, VFIO_DEVICE_RESET);
> + vcdev->vdev.io_ops->device_reset(&vcdev->vdev);
> }
>
> static void vfio_ccw_crw_read(VFIOCCWDevice *vcdev)
> diff --git a/hw/vfio/device.c b/hw/vfio/device.c
> index 4f11959633..e7d293710e 100644
> --- a/hw/vfio/device.c
> +++ b/hw/vfio/device.c
> @@ -578,9 +578,18 @@ int vfio_device_get_feature(VFIODevice *vbasedev,
> }
>
> /*
> - * Traditional ioctl() based io
> + * VFIODeviceIOOps implementation for the kernel VFIO backend.
> */
>
> +static int vfio_device_io_device_reset(VFIODevice *vbasedev)
> +{
> + int ret;
> +
> + ret = ioctl(vbasedev->fd, VFIO_DEVICE_RESET);
> +
> + return ret < 0 ? -errno : ret;
> +}
> +
> static int vfio_device_io_device_feature(VFIODevice *vbasedev,
> struct vfio_device_feature *feature)
> {
> @@ -659,7 +668,7 @@ static int vfio_device_io_region_write(VFIODevice *vbasedev, uint8_t index,
>
> static VFIODeviceIOOps vfio_device_io_ops_ioctl = {
> .capabilities = VFIO_IO_CAP_DMA_BUF,
> -
Please keep the empty line.
Thanks,
C.
> + .device_reset = vfio_device_io_device_reset,
> .device_feature = vfio_device_io_device_feature,
> .get_region_info = vfio_device_io_get_region_info,
> .get_irq_info = vfio_device_io_get_irq_info,
> diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c
> index ea6734823b..1941678e29 100644
> --- a/hw/vfio/migration.c
> +++ b/hw/vfio/migration.c
> @@ -142,7 +142,7 @@ int vfio_migration_set_state(VFIODevice *vbasedev,
> struct vfio_device_feature *feature = (struct vfio_device_feature *)buf;
> struct vfio_device_feature_mig_state *mig_state =
> (struct vfio_device_feature_mig_state *)feature->data;
> - int ret;
> + int ret, reset_ret;
> g_autofree char *error_prefix =
> g_strdup_printf("%s: Failed setting device state to %s.",
> vbasedev->name, mig_state_to_str(new_state));
> @@ -219,9 +219,10 @@ int vfio_migration_set_state(VFIODevice *vbasedev,
> return 0;
>
> reset_device:
> - if (ioctl(vbasedev->fd, VFIO_DEVICE_RESET)) {
> + reset_ret = vbasedev->io_ops->device_reset(vbasedev);
> + if (reset_ret) {
> hw_error("%s: Failed resetting device, err: %s", vbasedev->name,
> - strerror(errno));
> + strerror(-reset_ret));
> }
>
> vfio_migration_set_device_state(vbasedev, VFIO_DEVICE_STATE_RUNNING);
> diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
> index 428ab2f069..12f2974d19 100644
> --- a/hw/vfio/pci.c
> +++ b/hw/vfio/pci.c
> @@ -3785,7 +3785,7 @@ static void vfio_pci_reset(DeviceState *dev)
>
> if (vdev->vbasedev.reset_works &&
> (vdev->has_flr || !vdev->has_pm_reset) &&
> - !ioctl(vdev->vbasedev.fd, VFIO_DEVICE_RESET)) {
> + !vdev->vbasedev.io_ops->device_reset(&vdev->vbasedev)) {
> trace_vfio_pci_reset_flr(vdev->vbasedev.name);
> goto post_reset;
> }
> @@ -3797,7 +3797,7 @@ static void vfio_pci_reset(DeviceState *dev)
>
> /* If nothing else works and the device supports PM reset, use it */
> if (vdev->vbasedev.reset_works && vdev->has_pm_reset &&
> - !ioctl(vdev->vbasedev.fd, VFIO_DEVICE_RESET)) {
> + !vdev->vbasedev.io_ops->device_reset(&vdev->vbasedev)) {
> trace_vfio_pci_reset_pm(vdev->vbasedev.name);
> goto post_reset;
> }
> diff --git a/include/hw/vfio/vfio-device.h b/include/hw/vfio/vfio-device.h
> index 8472420d3f..a8fed550eb 100644
> --- a/include/hw/vfio/vfio-device.h
> +++ b/include/hw/vfio/vfio-device.h
> @@ -191,6 +191,17 @@ struct VFIODeviceIOOps {
> */
> uint64_t capabilities;
>
> + /**
> + * @device_reset
> + *
> + * Perform a device reset.
> + *
> + * @vdev: #VFIODevice to use
> + *
> + * Returns 0 on success or -errno.
> + */
> + int (*device_reset)(VFIODevice *vdev);
> +
> /**
> * @device_feature
> *
^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH 5/8] vfio: use io_ops abstraction for migration related ioctls
2026-09-03 13:08 [PATCH 0/8] vfio-user: live migration support Hugo Komatsu
` (3 preceding siblings ...)
2026-09-03 13:08 ` [PATCH 4/8] vfio: route device_reset " Hugo Komatsu
@ 2026-09-03 13:08 ` Hugo Komatsu
2026-09-08 14:47 ` John Levon
2026-09-03 13:08 ` [PATCH 6/8] vfio-user: enable DMA logging for dirty page tracking Hugo Komatsu
` (2 subsequent siblings)
7 siblings, 1 reply; 17+ messages in thread
From: Hugo Komatsu @ 2026-09-03 13:08 UTC (permalink / raw)
To: qemu-devel@nongnu.org
Cc: qemu-s390x@nongnu.org, Alex Williamson, Cédric Le Goater,
Cornelia Huck, Eric Farman, Halil Pasic, Jason Herne, John Levon,
Mark Cave-Ayland, Matthew Rosato, Pierrick Bouvier,
Rafael Castillo, Thanos Makatos, Tony Krowiak
The generic VFIO migration code in hw/vfio/migration.c issued read()
and qemu_file_get_to_fd() directly, hardcoding the kernel as the only
transport. Route these through the VFIODeviceIOOps table so an
alternate transport (vfio-user) can supply its own implementations.
Add three migration slots - get_precopy_info, mig_data_read, and
mig_data_write_from_file - and implement the kernel versions as thin
wrappers around the existing calls. vfio_query_precopy_size() and
vfio_save_block() now take a VFIODevice so they can reach the table.
This is a pure refactor: the kernel path is unchanged.
Co-authored-by: Rafael Castillo <rafael@trfs.me>
Change-Id: I714dab654034a285833f033a6a49218462c5f17d
Signed-off-by: Hugo Komatsu <hugo.komatsu@nutanix.com>
---
hw/vfio/device.c | 36 ++++++++++++++++++++++++++++++++
hw/vfio/migration.c | 37 ++++++++++++++++-----------------
include/hw/vfio/vfio-device.h | 39 +++++++++++++++++++++++++++++++++++
3 files changed, 93 insertions(+), 19 deletions(-)
diff --git a/hw/vfio/device.c b/hw/vfio/device.c
index e7d293710e..ff2d965740 100644
--- a/hw/vfio/device.c
+++ b/hw/vfio/device.c
@@ -30,8 +30,10 @@
#include "qemu/units.h"
#include "migration/cpr.h"
#include "migration/blocker.h"
+#include "migration/qemu-file.h"
#include "monitor/monitor.h"
#include "vfio-helpers.h"
+#include "vfio-migration-internal.h"
VFIODeviceList vfio_device_list =
QLIST_HEAD_INITIALIZER(vfio_device_list);
@@ -590,6 +592,37 @@ static int vfio_device_io_device_reset(VFIODevice *vbasedev)
return ret < 0 ? -errno : ret;
}
+static ssize_t vfio_device_io_mig_data_read(VFIODevice *vbasedev, void *buf,
+ size_t buf_size)
+{
+ VFIOMigration *migration = vbasedev->migration;
+ ssize_t ret;
+
+ ret = read(migration->data_fd, buf, buf_size);
+
+ return ret < 0 ? -errno : ret;
+}
+
+static int vfio_device_io_mig_data_write_from_file(VFIODevice *vbasedev,
+ QEMUFile *f,
+ size_t data_size)
+{
+ VFIOMigration *migration = vbasedev->migration;
+
+ return qemu_file_get_to_fd(f, migration->data_fd, data_size);
+}
+
+static int vfio_device_io_get_precopy_info(VFIODevice *vbasedev,
+ struct vfio_precopy_info *info)
+{
+ VFIOMigration *migration = vbasedev->migration;
+ int ret;
+
+ ret = ioctl(migration->data_fd, VFIO_MIG_GET_PRECOPY_INFO, info);
+
+ return ret < 0 ? -errno : ret;
+}
+
static int vfio_device_io_device_feature(VFIODevice *vbasedev,
struct vfio_device_feature *feature)
{
@@ -669,6 +702,9 @@ static int vfio_device_io_region_write(VFIODevice *vbasedev, uint8_t index,
static VFIODeviceIOOps vfio_device_io_ops_ioctl = {
.capabilities = VFIO_IO_CAP_DMA_BUF,
.device_reset = vfio_device_io_device_reset,
+ .mig_data_read = vfio_device_io_mig_data_read,
+ .mig_data_write_from_file = vfio_device_io_mig_data_write_from_file,
+ .get_precopy_info = vfio_device_io_get_precopy_info,
.device_feature = vfio_device_io_device_feature,
.get_region_info = vfio_device_io_get_region_info,
.get_irq_info = vfio_device_io_get_irq_info,
diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c
index 1941678e29..612f9193a5 100644
--- a/hw/vfio/migration.c
+++ b/hw/vfio/migration.c
@@ -247,10 +247,9 @@ vfio_migration_set_state_or_reset(VFIODevice *vbasedev,
static int vfio_load_buffer(QEMUFile *f, VFIODevice *vbasedev,
uint64_t data_size)
{
- VFIOMigration *migration = vbasedev->migration;
int ret;
- ret = qemu_file_get_to_fd(f, migration->data_fd, data_size);
+ ret = vbasedev->io_ops->mig_data_write_from_file(vbasedev, f, data_size);
trace_vfio_load_state_device_data(vbasedev->name, data_size, ret);
return ret;
@@ -369,20 +368,20 @@ static int vfio_query_stop_copy_size(VFIODevice *vbasedev)
return ret;
}
-static int vfio_query_precopy_size(VFIOMigration *migration)
+static int vfio_query_precopy_size(VFIODevice *vbasedev)
{
- VFIODevice *vbasedev = migration->vbasedev;
+ VFIOMigration *migration = vbasedev->migration;
struct vfio_precopy_info precopy = {
.argsz = sizeof(precopy),
};
bool reinit = false;
int ret = 0;
- if (ioctl(migration->data_fd, VFIO_MIG_GET_PRECOPY_INFO, &precopy)) {
+ ret = vbasedev->io_ops->get_precopy_info(vbasedev, &precopy);
+ if (ret) {
migration->precopy_init_size = 0;
migration->precopy_dirty_size = 0;
- ret = -errno;
- warn_report_once("VFIO device %s ioctl(VFIO_MIG_GET_PRECOPY_INFO) "
+ warn_report_once("VFIO device %s get_precopy_info "
"failed (%d)", vbasedev->name, ret);
} else {
bool overflow;
@@ -425,18 +424,20 @@ static int vfio_query_precopy_size(VFIOMigration *migration)
}
/* Returns the size of saved data on success and -errno on error */
-static ssize_t vfio_save_block(QEMUFile *f, VFIOMigration *migration)
+static ssize_t vfio_save_block(QEMUFile *f, VFIODevice *vbasedev)
{
+ VFIOMigration *migration = vbasedev->migration;
ssize_t data_size;
- data_size = read(migration->data_fd, migration->data_buffer,
- migration->data_buffer_size);
+ data_size = vbasedev->io_ops->mig_data_read(vbasedev,
+ migration->data_buffer,
+ migration->data_buffer_size);
if (data_size < 0) {
/*
* Pre-copy emptied all the device state for now. For more information,
* please refer to the Linux kernel VFIO uAPI.
*/
- if (errno == ENOMSG) {
+ if (data_size == -ENOMSG) {
if (!migration->event_precopy_empty_hit) {
trace_vfio_save_block_precopy_empty_hit(migration->vbasedev->name);
migration->event_precopy_empty_hit = true;
@@ -444,7 +445,7 @@ static ssize_t vfio_save_block(QEMUFile *f, VFIOMigration *migration)
return 0;
}
- return -errno;
+ return data_size;
}
if (data_size == 0) {
return 0;
@@ -599,7 +600,7 @@ static int vfio_save_setup(QEMUFile *f, void *opaque, Error **errp)
return ret;
}
- vfio_query_precopy_size(migration);
+ vfio_query_precopy_size(vbasedev);
if (migrate_switchover_ack() && !migrate_switchover_ack_legacy()) {
migration->request_switchover_ack = true;
}
@@ -662,12 +663,10 @@ static void vfio_save_cleanup(void *opaque)
static void vfio_state_pending_sync(VFIODevice *vbasedev)
{
- VFIOMigration *migration = vbasedev->migration;
-
vfio_query_stop_copy_size(vbasedev);
if (vfio_device_state_is_precopy(vbasedev)) {
- vfio_query_precopy_size(migration);
+ vfio_query_precopy_size(vbasedev);
}
}
@@ -738,7 +737,7 @@ static int vfio_save_iterate(QEMUFile *f, void *opaque)
migration->event_save_iterate_started = true;
}
- data_size = vfio_save_block(f, migration);
+ data_size = vfio_save_block(f, vbasedev);
if (data_size < 0) {
return data_size;
}
@@ -778,7 +777,7 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)
}
do {
- data_size = vfio_save_block(f, vbasedev->migration);
+ data_size = vfio_save_block(f, vbasedev);
if (data_size < 0) {
return data_size;
}
@@ -975,7 +974,7 @@ static void vfio_final_precopy_reinit_check(VFIODevice *vbasedev)
return;
}
- ret = vfio_query_precopy_size(migration);
+ ret = vfio_query_precopy_size(vbasedev);
if (ret) {
error_report("%s: Final precopy reinit check failed (err: %d)",
vbasedev->name, ret);
diff --git a/include/hw/vfio/vfio-device.h b/include/hw/vfio/vfio-device.h
index a8fed550eb..3b13cffbd3 100644
--- a/include/hw/vfio/vfio-device.h
+++ b/include/hw/vfio/vfio-device.h
@@ -202,6 +202,18 @@ struct VFIODeviceIOOps {
*/
int (*device_reset)(VFIODevice *vdev);
+ /**
+ * @get_precopy_info
+ *
+ * Get information about pre-copy migration data transfer progress.
+ *
+ * @vdev: #VFIODevice to use
+ * @info: pointer to struct vfio_precopy_info to fill in
+ *
+ * Returns 0 on success or -errno.
+ */
+ int (*get_precopy_info)(VFIODevice *vdev, struct vfio_precopy_info *info);
+
/**
* @device_feature
*
@@ -268,6 +280,33 @@ struct VFIODeviceIOOps {
int (*region_read)(VFIODevice *vdev, uint8_t nr, off_t off, uint32_t size,
void *data);
+ /**
+ * @mig_data_read
+ *
+ * Read migration data from the device.
+ *
+ * @vdev: #VFIODevice to use
+ * @buf: buffer to read into
+ * @buf_size: maximum number of bytes to read
+ *
+ * Returns the number of bytes read on success or -errno.
+ */
+ ssize_t (*mig_data_read)(VFIODevice *vdev, void *buf, size_t buf_size);
+
+ /**
+ * @mig_data_write_from_file
+ *
+ * Write migration data to the device, reading the data from a QEMUFile.
+ *
+ * @vdev: #VFIODevice to use
+ * @f: source to read the migration data from
+ * @data_size: number of bytes to write
+ *
+ * Returns 0 on success or -errno.
+ */
+ int (*mig_data_write_from_file)(VFIODevice *vdev, QEMUFile *f,
+ size_t data_size);
+
/**
* @region_write
*
--
2.39.3
^ permalink raw reply related [flat|nested] 17+ messages in thread* Re: [PATCH 5/8] vfio: use io_ops abstraction for migration related ioctls
2026-09-03 13:08 ` [PATCH 5/8] vfio: use io_ops abstraction for migration related ioctls Hugo Komatsu
@ 2026-09-08 14:47 ` John Levon
0 siblings, 0 replies; 17+ messages in thread
From: John Levon @ 2026-09-08 14:47 UTC (permalink / raw)
To: Hugo Komatsu
Cc: qemu-devel@nongnu.org, qemu-s390x@nongnu.org, Alex Williamson,
Cédric Le Goater, Cornelia Huck, Eric Farman, Halil Pasic,
Jason Herne, Mark Cave-Ayland, Matthew Rosato, Pierrick Bouvier,
Rafael Castillo, Thanos Makatos, Tony Krowiak
On Thu, Sep 03, 2026 at 02:08:47PM +0100, Hugo Komatsu wrote:
> @@ -247,10 +247,9 @@ vfio_migration_set_state_or_reset(VFIODevice *vbasedev,
> static int vfio_load_buffer(QEMUFile *f, VFIODevice *vbasedev,
> uint64_t data_size)
> {
> - VFIOMigration *migration = vbasedev->migration;
> int ret;
>
> - ret = qemu_file_get_to_fd(f, migration->data_fd, data_size);
> + ret = vbasedev->io_ops->mig_data_write_from_file(vbasedev, f, data_size);
I'm guessing Cédric will want the wrapper functions for all of these too;
otherwise, LGTM.
regards
john
^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH 6/8] vfio-user: enable DMA logging for dirty page tracking
2026-09-03 13:08 [PATCH 0/8] vfio-user: live migration support Hugo Komatsu
` (4 preceding siblings ...)
2026-09-03 13:08 ` [PATCH 5/8] vfio: use io_ops abstraction for migration related ioctls Hugo Komatsu
@ 2026-09-03 13:08 ` Hugo Komatsu
2026-09-08 14:52 ` John Levon
2026-09-03 13:08 ` [PATCH 7/8] vfio-user: implement mig_data read/write socket operations Hugo Komatsu
2026-09-03 13:08 ` [PATCH 8/8] vfio-user: enable live migration for pci devices Hugo Komatsu
7 siblings, 1 reply; 17+ messages in thread
From: Hugo Komatsu @ 2026-09-03 13:08 UTC (permalink / raw)
To: qemu-devel@nongnu.org
Cc: qemu-s390x@nongnu.org, Alex Williamson, Cédric Le Goater,
Cornelia Huck, Eric Farman, Halil Pasic, Jason Herne, John Levon,
Mark Cave-Ayland, Matthew Rosato, Pierrick Bouvier,
Rafael Castillo, Thanos Makatos, Tony Krowiak
Forward the DMA logging features to the server so QEMU can perform
device dirty page tracking. START/STOP and the feature probe map onto
the existing device_feature message path.
REPORT needs bespoke handling: the kernel struct
vfio_device_feature_dma_logging_report carries the dirty bitmap by
pointer, whereas over vfio-user the bitmap travels inline after the
report header. Translate to the wire layout, size argsz to include
the bitmap, and copy the returned bitmap back into the caller buffer.
The server only tracks dirty pages for ranges it has DMA-mapped. A
range it cannot translate (e.g. firmware/MMIO regions never mapped to
the device) cannot have been dirtied by device DMA, so such a report
is treated as clean rather than failing the migration.
Signed-off-by: Hugo Komatsu <hugo.komatsu@nutanix.com>
---
hw/vfio-user/device.c | 77 +++++++++++++++++++++++++++++++++++++++
hw/vfio-user/trace-events | 1 +
2 files changed, 78 insertions(+)
diff --git a/hw/vfio-user/device.c b/hw/vfio-user/device.c
index 1a01e748cd..078fd8f2a4 100644
--- a/hw/vfio-user/device.c
+++ b/hw/vfio-user/device.c
@@ -74,6 +74,66 @@ void vfio_user_device_reset(VFIOUserProxy *proxy)
}
}
+/*
+ * Fetches a dirty page bitmap from the server for the specified DMA logging
+ * range. Over vfio-user, the bitmap is appended directly to the wire message
+ * instead of being passed by pointer.
+ */
+static int vfio_user_dma_logging_report(VFIODevice *vbasedev,
+ struct vfio_device_feature *feature)
+{
+ VFIOUserProxy *proxy = vbasedev->proxy;
+ struct vfio_device_feature_dma_logging_report *report =
+ (struct vfio_device_feature_dma_logging_report *)feature->data;
+ uint32_t report_hdr_size =
+ offsetof(struct vfio_device_feature_dma_logging_report, bitmap);
+ void *bitmap = (void *)(uintptr_t)report->bitmap;
+ uint64_t nr_pages, bitmap_size;
+ uint32_t wire_argsz, msg_size;
+ g_autofree VFIOUserDeviceFeature *msgp = NULL;
+ Error *local_err = NULL;
+
+ if (report->page_size == 0) {
+ return -EINVAL;
+ }
+
+ nr_pages = DIV_ROUND_UP(report->length, report->page_size);
+ bitmap_size = ROUND_UP(nr_pages, 64) / 8;
+
+ wire_argsz = sizeof(struct vfio_device_feature) + report_hdr_size +
+ bitmap_size;
+ msg_size = sizeof(VFIOUserHdr) + wire_argsz;
+
+ if (msg_size > proxy->max_xfer_size) {
+ error_report("%s: DMA logging report size exceeds max_xfer_size",
+ vbasedev->name);
+ return -EINVAL;
+ }
+
+ msgp = g_malloc0(msg_size);
+ vfio_user_request_msg(&msgp->hdr, VFIO_USER_DEVICE_FEATURE, msg_size, 0);
+ msgp->argsz = wire_argsz;
+ msgp->flags = feature->flags;
+ memcpy(msgp->data, report, report_hdr_size);
+
+ if (!vfio_user_send_wait(proxy, &msgp->hdr, NULL, msg_size, &local_err)) {
+ error_prepend(&local_err, "%s: ", __func__);
+ error_report_err(local_err);
+ return -EFAULT;
+ }
+
+ if (msgp->hdr.flags & VFIO_USER_ERROR) {
+ error_report("%s: server returned error %d", __func__,
+ msgp->hdr.error_reply);
+ return -msgp->hdr.error_reply;
+ }
+
+ memcpy(bitmap, msgp->data + report_hdr_size, bitmap_size);
+
+ trace_vfio_user_dma_logging_report(feature->flags, bitmap_size);
+ return 0;
+}
+
static int
vfio_user_device_io_device_feature(VFIODevice *vbasedev,
struct vfio_device_feature *feature)
@@ -81,12 +141,29 @@ vfio_user_device_io_device_feature(VFIODevice *vbasedev,
g_autofree VFIOUserDeviceFeature *msgp = NULL;
VFIOUserProxy *proxy = vbasedev->proxy;
Error *local_err = NULL;
+ uint16_t feat = feature->flags & VFIO_DEVICE_FEATURE_MASK;
int size;
+ /*
+ * DMA (dirty-page) logging report needs bespoke handling. The kernel
+ * struct vfio_device_feature_dma_logging_report carries the dirty bitmap
+ * by pointer, but over vfio-user the bitmap travels inline after the
+ * report header. Translate to the wire layout, size argsz to include the
+ * bitmap, send, and copy the returned bitmap back into the caller buffer.
+ *
+ * START/STOP (and the PROBE) fall through to the generic path below: the
+ * server only reads page_size (same offset in both layouts) and ignores
+ * the ranges.
+ */
+ if (feat == VFIO_DEVICE_FEATURE_DMA_LOGGING_REPORT) {
+ return vfio_user_dma_logging_report(vbasedev, feature);
+ }
+
if (__builtin_add_overflow(feature->argsz, sizeof(VFIOUserHdr), &size)) {
error_printf("vfio_user_device_io_device_feature argsz too large\n");
return -E2BIG;
}
+
if (size > proxy->max_xfer_size) {
error_printf("vfio_user_device_io_device_feature argsz too large\n");
return -E2BIG;
diff --git a/hw/vfio-user/trace-events b/hw/vfio-user/trace-events
index 503e1c82d4..35341994c5 100644
--- a/hw/vfio-user/trace-events
+++ b/hw/vfio-user/trace-events
@@ -8,6 +8,7 @@ vfio_user_dma_unmap(uint64_t iova, uint64_t size, uint32_t flags, bool async_op
# device.c
vfio_user_device_io_device_feature(uint32_t argsz, uint32_t flags) " argsz 0x%x flags 0x%x"
+vfio_user_dma_logging_report(uint32_t flags, uint64_t bitmap_size) " flags 0x%"PRIx32" bitmap_size 0x%"PRIx64
vfio_user_get_info(uint32_t nregions, uint32_t nirqs) " #regions %d #irqs %d"
vfio_user_get_irq_info(uint32_t index, uint32_t flags, uint32_t count) " index %d flags 0x%x count %d"
vfio_user_set_irqs(uint32_t index, uint32_t start, uint32_t count, uint32_t flags) " index %d start %d count %d flags 0x%x"
--
2.39.3
^ permalink raw reply related [flat|nested] 17+ messages in thread* Re: [PATCH 6/8] vfio-user: enable DMA logging for dirty page tracking
2026-09-03 13:08 ` [PATCH 6/8] vfio-user: enable DMA logging for dirty page tracking Hugo Komatsu
@ 2026-09-08 14:52 ` John Levon
0 siblings, 0 replies; 17+ messages in thread
From: John Levon @ 2026-09-08 14:52 UTC (permalink / raw)
To: Hugo Komatsu
Cc: qemu-devel@nongnu.org, qemu-s390x@nongnu.org, Alex Williamson,
Cédric Le Goater, Cornelia Huck, Eric Farman, Halil Pasic,
Jason Herne, Mark Cave-Ayland, Matthew Rosato, Pierrick Bouvier,
Rafael Castillo, Thanos Makatos, Tony Krowiak
On Thu, Sep 03, 2026 at 02:08:49PM +0100, Hugo Komatsu wrote:
> Forward the DMA logging features to the server so QEMU can perform
> device dirty page tracking. START/STOP and the feature probe map onto
> the existing device_feature message path.
>
> REPORT needs bespoke handling: the kernel struct
> vfio_device_feature_dma_logging_report carries the dirty bitmap by
> pointer, whereas over vfio-user the bitmap travels inline after the
> report header. Translate to the wire layout, size argsz to include
> the bitmap, and copy the returned bitmap back into the caller buffer.
>
> The server only tracks dirty pages for ranges it has DMA-mapped. A
> range it cannot translate (e.g. firmware/MMIO regions never mapped to
> the device) cannot have been dirtied by device DMA, so such a report
> is treated as clean rather than failing the migration.
>
> Signed-off-by: Hugo Komatsu <hugo.komatsu@nutanix.com>
Reviewed-by: John Levon <john.levon@nutanix.com>
^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH 7/8] vfio-user: implement mig_data read/write socket operations
2026-09-03 13:08 [PATCH 0/8] vfio-user: live migration support Hugo Komatsu
` (5 preceding siblings ...)
2026-09-03 13:08 ` [PATCH 6/8] vfio-user: enable DMA logging for dirty page tracking Hugo Komatsu
@ 2026-09-03 13:08 ` Hugo Komatsu
2026-09-08 14:55 ` John Levon
2026-09-03 13:08 ` [PATCH 8/8] vfio-user: enable live migration for pci devices Hugo Komatsu
7 siblings, 1 reply; 17+ messages in thread
From: Hugo Komatsu @ 2026-09-03 13:08 UTC (permalink / raw)
To: qemu-devel@nongnu.org
Cc: qemu-s390x@nongnu.org, Alex Williamson, Cédric Le Goater,
Cornelia Huck, Eric Farman, Halil Pasic, Jason Herne, John Levon,
Mark Cave-Ayland, Matthew Rosato, Pierrick Bouvier,
Rafael Castillo, Thanos Makatos, Tony Krowiak
Implement the remaining vfio-user (socket) migration io_ops:
device_reset, get_precopy_info, mig_data_read, and
mig_data_write_from_file.
Add the wire definitions (command numbers and message layouts) to
hw/vfio-user/protocol.h, mirroring the kernel structs so feature blobs
can be copied straight onto the wire.
Co-authored-by: Rafael Castillo <rafael@trfs.me>
Signed-off-by: Hugo Komatsu <hugo.komatsu@nutanix.com>
---
hw/vfio-user/device.c | 118 +++++++++++++++++++++++++++++++++++++-
hw/vfio-user/protocol.h | 19 ++++++
hw/vfio-user/trace-events | 2 +
hw/vfio/migration.c | 6 +-
4 files changed, 140 insertions(+), 5 deletions(-)
diff --git a/hw/vfio-user/device.c b/hw/vfio-user/device.c
index 078fd8f2a4..6b767e94d3 100644
--- a/hw/vfio-user/device.c
+++ b/hw/vfio-user/device.c
@@ -11,6 +11,7 @@
#include "qemu/error-report.h"
#include "qemu/lockable.h"
#include "qemu/thread.h"
+#include "migration/qemu-file.h"
#include "hw/vfio-user/device.h"
#include "hw/vfio-user/trace.h"
@@ -592,15 +593,126 @@ static int vfio_user_device_io_region_write(VFIODevice *vbasedev, uint8_t index,
return ret;
}
+static ssize_t vfio_user_device_io_mig_data_read(VFIODevice *vbasedev,
+ void *buf,
+ size_t buf_size)
+{
+ g_autofree VFIOUserMigData *msgp = NULL;
+ VFIOUserProxy *proxy = vbasedev->proxy;
+ Error *local_err = NULL;
+ /*
+ * We don't need to iterate in chunks here. We clamp the requested
+ * size to the maximum socket payload and return a "short read"
+ * (fewer bytes than buf_size). QEMU's generic migration framework
+ * handles short reads perfectly, and will automatically call this
+ * function again in a loop to fetch the remaining data.
+ */
+ uint32_t max_payload = proxy->max_xfer_size - sizeof(VFIOUserMigData);
+ uint32_t read_size = MIN(buf_size, max_payload);
+ uint32_t msg_size = sizeof(VFIOUserMigData) + read_size;
+
+ /* Clamp the requested size to the maximum socket payload */
+ msgp = g_malloc0(msg_size);
+
+ vfio_user_request_msg(&msgp->hdr, VFIO_USER_MIG_DATA_READ,
+ sizeof(VFIOUserMigData), 0);
+ msgp->argsz = msg_size - sizeof(VFIOUserHdr);
+ msgp->size = read_size;
+
+ if (!vfio_user_send_wait(proxy, &msgp->hdr, NULL, msg_size, &local_err)) {
+ error_prepend(&local_err, "%s: ", __func__);
+ error_report_err(local_err);
+ return -EFAULT;
+ }
+
+ if (msgp->hdr.flags & VFIO_USER_ERROR) {
+ error_report("%s: server returned error %d", __func__,
+ msgp->hdr.error_reply);
+ return -msgp->hdr.error_reply;
+ }
+
+ /* Bounds check: Ensure server didn't overflow our buffer */
+ if (msgp->size > buf_size) {
+ error_report("%s: server sent more data than requested", __func__);
+ return -EINVAL;
+ }
+
+ memcpy(buf, msgp->data, msgp->size);
+ trace_vfio_user_mig_data_read(msgp->size);
+
+ return msgp->size;
+}
+
+static int vfio_user_device_io_mig_data_write_from_file(VFIODevice *vbasedev,
+ QEMUFile *f,
+ size_t data_size)
+{
+ VFIOUserProxy *proxy = vbasedev->proxy;
+ uint32_t max_payload = proxy->max_xfer_size - sizeof(VFIOUserMigData);
+ size_t remaining = data_size;
+ g_autofree VFIOUserMigData *msgp =
+ g_malloc0(sizeof(VFIOUserMigData) + max_payload);
+
+ while (remaining > 0) {
+ Error *local_err = NULL;
+ uint32_t chunk_size = MIN(remaining, max_payload);
+ uint32_t msg_size = sizeof(VFIOUserMigData) + chunk_size;
+
+ vfio_user_request_msg(&msgp->hdr, VFIO_USER_MIG_DATA_WRITE,
+ msg_size, 0);
+ msgp->argsz = msg_size - sizeof(VFIOUserHdr);
+ msgp->size = chunk_size;
+
+ if (qemu_get_buffer(f, msgp->data, chunk_size) != chunk_size) {
+ error_report("%s: failed to read migration data from stream",
+ __func__);
+ return -EINVAL;
+ }
+
+ if (!vfio_user_send_wait(proxy, &msgp->hdr, NULL, msg_size,
+ &local_err)) {
+ error_prepend(&local_err, "%s: ", __func__);
+ error_report_err(local_err);
+ return -EFAULT;
+ }
+
+ if (msgp->hdr.flags & VFIO_USER_ERROR) {
+ error_report("%s: server returned error %d", __func__,
+ msgp->hdr.error_reply);
+ return -msgp->hdr.error_reply;
+ }
+
+ trace_vfio_user_mig_data_write(msgp->size);
+ remaining -= chunk_size;
+ }
+
+ return 0;
+}
+
+static int vfio_user_device_io_device_reset(VFIODevice *vbasedev)
+{
+ vfio_user_device_reset(vbasedev->proxy);
+ return 0;
+}
+
+static int vfio_user_device_io_get_precopy_info(VFIODevice *vbasedev,
+ struct vfio_precopy_info *info)
+{
+ return -ENOTSUP;
+}
+
/*
* Socket-based io_ops
*/
VFIODeviceIOOps vfio_user_device_io_ops_sock = {
.device_feature = vfio_user_device_io_device_feature,
.get_region_info = vfio_user_device_io_get_region_info,
- .get_irq_info = vfio_user_device_io_get_irq_info,
- .set_irqs = vfio_user_device_io_set_irqs,
.region_read = vfio_user_device_io_region_read,
.region_write = vfio_user_device_io_region_write,
-
+ .get_irq_info = vfio_user_device_io_get_irq_info,
+ .set_irqs = vfio_user_device_io_set_irqs,
+ .device_reset = vfio_user_device_io_device_reset,
+ .get_precopy_info = vfio_user_device_io_get_precopy_info,
+ .mig_data_read = vfio_user_device_io_mig_data_read,
+ .mig_data_write_from_file = vfio_user_device_io_mig_data_write_from_file,
};
diff --git a/hw/vfio-user/protocol.h b/hw/vfio-user/protocol.h
index c478d1353f..ed32d73051 100644
--- a/hw/vfio-user/protocol.h
+++ b/hw/vfio-user/protocol.h
@@ -41,6 +41,9 @@ enum vfio_user_command {
VFIO_USER_DIRTY_PAGES = 14,
VFIO_USER_REGION_WRITE_MULTI = 15,
VFIO_USER_DEVICE_FEATURE = 16,
+ VFIO_USER_MIG_DATA_READ = 17,
+ VFIO_USER_MIG_DATA_WRITE = 18,
+ VFIO_USER_GET_PRECOPY_INFO = 19,
VFIO_USER_MAX,
};
@@ -251,4 +254,20 @@ typedef struct {
char data[];
} VFIOUserDeviceFeature;
+/* imported from struct vfio_mig_data */
+typedef struct {
+ VFIOUserHdr hdr;
+ uint32_t argsz;
+ uint32_t size;
+ uint8_t data[];
+} VFIOUserMigData;
+
+/* imported from struct vfio_precopy_info */
+typedef struct {
+ VFIOUserHdr hdr;
+ uint32_t argsz;
+ uint32_t flags;
+ uint64_t initial_bytes;
+ uint64_t dirty_bytes;
+} VFIOUserGetPrecopyInfo;
#endif /* VFIO_USER_PROTOCOL_H */
diff --git a/hw/vfio-user/trace-events b/hw/vfio-user/trace-events
index 35341994c5..51abd4d738 100644
--- a/hw/vfio-user/trace-events
+++ b/hw/vfio-user/trace-events
@@ -14,6 +14,8 @@ vfio_user_get_irq_info(uint32_t index, uint32_t flags, uint32_t count) " index %
vfio_user_set_irqs(uint32_t index, uint32_t start, uint32_t count, uint32_t flags) " index %d start %d count %d flags 0x%x"
vfio_user_get_region_info(uint32_t index, uint32_t flags, uint64_t size) " index %d flags 0x%x size 0x%"PRIx64
vfio_user_region_rw(uint32_t region, uint64_t off, uint32_t count) " region %d offset 0x%"PRIx64" count %d"
+vfio_user_mig_data_read(uint32_t size) " size %d"
+vfio_user_mig_data_write(uint32_t size) " size %d"
# proxy.c
vfio_user_recv_hdr(const char *name, uint16_t id, uint16_t cmd, uint32_t size, uint32_t flags) " (%s) id 0x%x cmd 0x%x size 0x%x flags 0x%x"
diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c
index 612f9193a5..d12d7a2f03 100644
--- a/hw/vfio/migration.c
+++ b/hw/vfio/migration.c
@@ -313,8 +313,10 @@ static void vfio_migration_cleanup(VFIODevice *vbasedev)
{
VFIOMigration *migration = vbasedev->migration;
- close(migration->data_fd);
- migration->data_fd = -1;
+ if (migration->data_fd != -1) {
+ close(migration->data_fd);
+ migration->data_fd = -1;
+ }
}
static bool vfio_migration_check_overflow(VFIODevice *vbasedev, uint64_t size,
--
2.39.3
^ permalink raw reply related [flat|nested] 17+ messages in thread* Re: [PATCH 7/8] vfio-user: implement mig_data read/write socket operations
2026-09-03 13:08 ` [PATCH 7/8] vfio-user: implement mig_data read/write socket operations Hugo Komatsu
@ 2026-09-08 14:55 ` John Levon
0 siblings, 0 replies; 17+ messages in thread
From: John Levon @ 2026-09-08 14:55 UTC (permalink / raw)
To: Hugo Komatsu
Cc: qemu-devel@nongnu.org, qemu-s390x@nongnu.org, Alex Williamson,
Cédric Le Goater, Cornelia Huck, Eric Farman, Halil Pasic,
Jason Herne, Mark Cave-Ayland, Matthew Rosato, Pierrick Bouvier,
Rafael Castillo, Thanos Makatos, Tony Krowiak
On Thu, Sep 03, 2026 at 02:08:51PM +0100, Hugo Komatsu wrote:
> Implement the remaining vfio-user (socket) migration io_ops:
> device_reset, get_precopy_info, mig_data_read, and
> mig_data_write_from_file.
>
> Add the wire definitions (command numbers and message layouts) to
> hw/vfio-user/protocol.h, mirroring the kernel structs so feature blobs
> can be copied straight onto the wire.
Reviewed-by: John Levon <john.levon@nutanix.com>
^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH 8/8] vfio-user: enable live migration for pci devices
2026-09-03 13:08 [PATCH 0/8] vfio-user: live migration support Hugo Komatsu
` (6 preceding siblings ...)
2026-09-03 13:08 ` [PATCH 7/8] vfio-user: implement mig_data read/write socket operations Hugo Komatsu
@ 2026-09-03 13:08 ` Hugo Komatsu
2026-09-08 14:58 ` John Levon
7 siblings, 1 reply; 17+ messages in thread
From: Hugo Komatsu @ 2026-09-03 13:08 UTC (permalink / raw)
To: qemu-devel@nongnu.org
Cc: qemu-s390x@nongnu.org, Alex Williamson, Cédric Le Goater,
Cornelia Huck, Eric Farman, Halil Pasic, Jason Herne, John Levon,
Mark Cave-Ayland, Matthew Rosato, Pierrick Bouvier,
Rafael Castillo, Thanos Makatos, Tony Krowiak
Enroll vfio-user PCI devices in QEMU's migration machinery. Export the
kernel vfio-pci config save/load helpers (vfio_pci_save_config and
vfio_pci_load_config) and reuse them for vfio-user instead of
duplicating the logic. Call vfio_migration_realize() from
vfio_user_pci_realize() so the device registers its save/load handlers.
Expose the enable-migration on/off/auto property, mirroring kernel
vfio-pci, so migration can be explicitly opted into. Peer-to-Peer (P2P)
migration states are not supported by the libvfio-user server, so P2P
migration support is not included.
Device state pre-copy is not supported over vfio-user yet. Strip the
PRE_COPY bit from the advertised migration flags so the generic
migration code never enters the pre-copy path, forcing a plain
stop-and-copy migration.
Co-authored-by: Rafael Castillo <rafael@trfs.me>
Change-Id: I110499935b883d9436f6335e53bc3783cf530379
Signed-off-by: Hugo Komatsu <hugo.komatsu@nutanix.com>
---
hw/vfio-user/device.c | 11 +++++++++++
hw/vfio-user/pci.c | 13 ++++++++++---
hw/vfio/pci.c | 27 +++++++++++++++++----------
hw/vfio/pci.h | 3 +++
4 files changed, 41 insertions(+), 13 deletions(-)
diff --git a/hw/vfio-user/device.c b/hw/vfio-user/device.c
index 6b767e94d3..a0c23e1a68 100644
--- a/hw/vfio-user/device.c
+++ b/hw/vfio-user/device.c
@@ -197,6 +197,17 @@ vfio_user_device_io_device_feature(VFIODevice *vbasedev,
memcpy(feature, &msgp->argsz, feature->argsz);
+ /*
+ * Pre-copy is not supported over vfio-user yet. Strip the PRE_COPY bit
+ * from the advertised migration flags so the generic migration code never
+ * enters the pre-copy path, forcing a plain stop-and-copy migration.
+ */
+ if (feat == VFIO_DEVICE_FEATURE_MIGRATION) {
+ struct vfio_device_feature_migration *mig =
+ (struct vfio_device_feature_migration *)feature->data;
+ mig->flags &= ~VFIO_MIGRATION_PRE_COPY;
+ }
+
trace_vfio_user_device_io_device_feature(msgp->argsz, msgp->flags);
return 0;
diff --git a/hw/vfio-user/pci.c b/hw/vfio-user/pci.c
index e7573d4a9f..b9f65e78ce 100644
--- a/hw/vfio-user/pci.c
+++ b/hw/vfio-user/pci.c
@@ -13,6 +13,7 @@
#include "hw/core/qdev-properties.h"
#include "hw/vfio/pci.h"
+#include "hw/vfio/vfio-migration-internal.h"
#include "hw/vfio-user/device.h"
#include "hw/vfio-user/proxy.h"
@@ -239,9 +240,8 @@ static VFIODeviceOps vfio_user_pci_ops = {
.vfio_compute_needs_reset = vfio_user_compute_needs_reset,
.vfio_eoi = vfio_pci_intx_eoi,
.vfio_get_object = vfio_user_pci_get_object,
- /* No live migration support yet. */
- .vfio_save_config = NULL,
- .vfio_load_config = NULL,
+ .vfio_save_config = vfio_pci_save_config,
+ .vfio_load_config = vfio_pci_load_config,
};
static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
@@ -343,6 +343,11 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
goto out_teardown;
}
+ if (!vfio_migration_realize(vbasedev, errp)) {
+ vfio_pci_interrupt_teardown(vdev);
+ goto out_teardown;
+ }
+
vfio_pci_register_err_notifier(vdev);
vfio_pci_register_req_notifier(vdev);
@@ -426,6 +431,8 @@ static const Property vfio_user_pci_properties[] = {
DEFINE_PROP_BOOL("x-send-queued", VFIOUserPCIDevice, send_queued, false),
DEFINE_PROP_UINT32("x-msg-timeout", VFIOUserPCIDevice, wait_time, 5000),
DEFINE_PROP_BOOL("x-no-posted-writes", VFIOUserPCIDevice, no_post, false),
+ DEFINE_PROP_ON_OFF_AUTO("enable-migration", VFIOPCIDevice,
+ vbasedev.enable_migration, ON_OFF_AUTO_AUTO),
};
static void vfio_user_pci_set_socket(Object *obj, Visitor *v, const char *name,
diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
index 12f2974d19..5363288c61 100644
--- a/hw/vfio/pci.c
+++ b/hw/vfio/pci.c
@@ -2996,7 +2996,7 @@ static const VMStateDescription vmstate_vfio_pci_config = {
}
};
-static int vfio_pci_save_config(VFIODevice *vbasedev, QEMUFile *f, Error **errp)
+int vfio_pci_save_config(VFIODevice *vbasedev, QEMUFile *f, Error **errp)
{
VFIOPCIDevice *vdev = container_of(vbasedev, VFIOPCIDevice, vbasedev);
@@ -3004,7 +3004,7 @@ static int vfio_pci_save_config(VFIODevice *vbasedev, QEMUFile *f, Error **errp)
errp);
}
-static int vfio_pci_load_config(VFIODevice *vbasedev, QEMUFile *f)
+int vfio_pci_load_config(VFIODevice *vbasedev, QEMUFile *f)
{
VFIOPCIDevice *vdev = container_of(vbasedev, VFIOPCIDevice, vbasedev);
PCIDevice *pdev = PCI_DEVICE(vdev);
@@ -3570,6 +3570,20 @@ bool vfio_pci_interrupt_setup(VFIOPCIDevice *vdev, Error **errp)
return true;
}
+void vfio_pci_interrupt_teardown(VFIOPCIDevice *vdev)
+{
+ pci_device_set_intx_routing_notifier(PCI_DEVICE(vdev), NULL);
+ if (vdev->irqchip_change_notifier.notify) {
+ kvm_irqchip_remove_change_notifier(&vdev->irqchip_change_notifier);
+ vdev->irqchip_change_notifier.notify = NULL;
+ }
+ vfio_disable_interrupts(vdev);
+ if (vdev->intx.mmap_timer) {
+ timer_free(vdev->intx.mmap_timer);
+ vdev->intx.mmap_timer = NULL;
+ }
+}
+
static void vfio_pci_realize(PCIDevice *pdev, Error **errp)
{
ERRP_GUARD();
@@ -3745,14 +3759,7 @@ static void vfio_exitfn(PCIDevice *pdev)
vfio_display_exit(vdev);
vfio_unregister_req_notifier(vdev);
vfio_unregister_err_notifier(vdev);
- pci_device_set_intx_routing_notifier(pdev, NULL);
- if (vdev->irqchip_change_notifier.notify) {
- kvm_irqchip_remove_change_notifier(&vdev->irqchip_change_notifier);
- }
- vfio_disable_interrupts(vdev);
- if (vdev->intx.mmap_timer) {
- timer_free(vdev->intx.mmap_timer);
- }
+ vfio_pci_interrupt_teardown(vdev);
vfio_pci_teardown_msi(vdev);
vfio_pci_disable_rp_atomics(vdev);
vfio_pci_bars_exit(vdev);
diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h
index c9ab949870..9298e0e996 100644
--- a/hw/vfio/pci.h
+++ b/hw/vfio/pci.h
@@ -221,6 +221,8 @@ bool vfio_pci_intx_enable(VFIOPCIDevice *vdev, Error **errp);
void vfio_pci_intx_set_handler(VFIOPCIDevice *vdev, bool enable);
void vfio_pci_msix_set_notifiers(VFIOPCIDevice *vdev);
void vfio_pci_msi_set_handler(VFIOPCIDevice *vdev, int nr, bool enable);
+int vfio_pci_save_config(VFIODevice *vbasedev, QEMUFile *f, Error **errp);
+int vfio_pci_load_config(VFIODevice *vbasedev, QEMUFile *f);
uint32_t vfio_pci_read_config(PCIDevice *pdev, uint32_t addr, int len);
void vfio_pci_write_config(PCIDevice *pdev,
@@ -284,6 +286,7 @@ bool vfio_pci_add_capabilities(VFIOPCIDevice *vdev, Error **errp);
void vfio_pci_config_register_vga(VFIOPCIDevice *vdev);
bool vfio_pci_config_setup(VFIOPCIDevice *vdev, Error **errp);
bool vfio_pci_interrupt_setup(VFIOPCIDevice *vdev, Error **errp);
+void vfio_pci_interrupt_teardown(VFIOPCIDevice *vdev);
void vfio_pci_intx_eoi(VFIODevice *vbasedev);
void vfio_pci_put_device(VFIOPCIDevice *vdev);
bool vfio_pci_populate_device(VFIOPCIDevice *vdev, Error **errp);
--
2.39.3
^ permalink raw reply related [flat|nested] 17+ messages in thread* Re: [PATCH 8/8] vfio-user: enable live migration for pci devices
2026-09-03 13:08 ` [PATCH 8/8] vfio-user: enable live migration for pci devices Hugo Komatsu
@ 2026-09-08 14:58 ` John Levon
0 siblings, 0 replies; 17+ messages in thread
From: John Levon @ 2026-09-08 14:58 UTC (permalink / raw)
To: Hugo Komatsu
Cc: qemu-devel@nongnu.org, qemu-s390x@nongnu.org, Alex Williamson,
Cédric Le Goater, Cornelia Huck, Eric Farman, Halil Pasic,
Jason Herne, Mark Cave-Ayland, Matthew Rosato, Pierrick Bouvier,
Rafael Castillo, Thanos Makatos, Tony Krowiak
On Thu, Sep 03, 2026 at 02:08:52PM +0100, Hugo Komatsu wrote:
> Enroll vfio-user PCI devices in QEMU's migration machinery. Export the
> kernel vfio-pci config save/load helpers (vfio_pci_save_config and
> vfio_pci_load_config) and reuse them for vfio-user instead of
> duplicating the logic. Call vfio_migration_realize() from
> vfio_user_pci_realize() so the device registers its save/load handlers.
>
> Expose the enable-migration on/off/auto property, mirroring kernel
> vfio-pci, so migration can be explicitly opted into. Peer-to-Peer (P2P)
> migration states are not supported by the libvfio-user server, so P2P
> migration support is not included.
>
> Device state pre-copy is not supported over vfio-user yet. Strip the
> PRE_COPY bit from the advertised migration flags so the generic
> migration code never enters the pre-copy path, forcing a plain
> stop-and-copy migration.
>
> Co-authored-by: Rafael Castillo <rafael@trfs.me>
> Change-Id: I110499935b883d9436f6335e53bc3783cf530379
Please drop the Change-Id that snuck in.
Otherwise
Reviewed-by: John Levon <john.levon@nutanix.com>
^ permalink raw reply [flat|nested] 17+ messages in thread