[PATCH v3 0/4] qapi/vfio: Add VFIO migration QAPI event

[PATCH v3 0/4] qapi/vfio: Add VFIO migration QAPI event

[Avihai Horon]

Hello,

This series adds a new QAPI event for VFIO device migration state 
change. This event will be emitted when a VFIO device changes its
state, for example, during migration or when stopping/starting the
guest.

This event can be used by management applications to get updates on the
current state of the VFIO device for their own purposes.

A new per VFIO device capability, "migration-events", is added so events
can be enabled only for the required devices. It is disabled by default.

Feedback/comments are appreciated,

Thanks.

Changes from v2 [2]:
* Added assert for vbasedev->ops->vfio_get_object and obj. (Cedric)
* Renamed set_state() to vfio_migration_set_device_state(). (Cedric)
* Enhanced tracing of device state change. (Cedric)
* Added Cedric's R-b.

Changes from v1 [1]:
* Added more info to patch #1 commit mesasge. (Markus)
* Renamed VFIODeviceMigState to VfioMigrationState and
  VFIO_DEVICE_MIG_STATE_CHANGED to VFIO_MIGRATION. (Joao, Markus)
* Added qom-path and qdev id to VFIO_MIGRATION event data. (Markus)
* Handled no-op state transitions in vfio_migration_set_state().
  (Cedric)
* Added helper to set VFIO state and emit VFIO event. (Peter)

[1]
https://lore.kernel.org/qemu-devel/20240430051621.19597-1-avihaih@nvidia.com/

[2]
https://lore.kernel.org/qemu-devel/20240509090954.16447-1-avihaih@nvidia.com/

Avihai Horon (4):
  qapi/vfio: Add VFIO migration QAPI event
  vfio/migration: Emit VFIO migration QAPI event
  vfio/migration: Don't emit STOP_COPY VFIO migration QAPI event twice
  vfio/migration: Enhance VFIO migration state tracing

 MAINTAINERS                   |  1 +
 qapi/qapi-schema.json         |  1 +
 qapi/vfio.json                | 67 +++++++++++++++++++++++++++++++++
 include/hw/vfio/vfio-common.h |  1 +
 hw/vfio/migration.c           | 71 ++++++++++++++++++++++++++++++++---
 hw/vfio/pci.c                 |  2 +
 hw/vfio/trace-events          |  3 +-
 qapi/meson.build              |  1 +
 8 files changed, 141 insertions(+), 6 deletions(-)
 create mode 100644 qapi/vfio.json

-- 
2.26.3

[PATCH v3 1/4] qapi/vfio: Add VFIO migration QAPI event

[Avihai Horon]

Add a new QAPI event for VFIO migration. This event will be emitted when
a VFIO device changes its migration state, for example, during migration
or when stopping/starting the guest.

This event can be used by management applications to get updates on the
current state of the VFIO device for their own purposes.

Note that this new event is introduced since VFIO devices have a unique
set of migration states which cannot be described as accurately by other
existing events such as run state or migration status.

Signed-off-by: Avihai Horon <avihaih@nvidia.com>
Reviewed-by: Cédric Le Goater <clg@redhat.com>
---
 MAINTAINERS           |  1 +
 qapi/qapi-schema.json |  1 +
 qapi/vfio.json        | 67 +++++++++++++++++++++++++++++++++++++++++++
 qapi/meson.build      |  1 +
 4 files changed, 70 insertions(+)
 create mode 100644 qapi/vfio.json

diff --git a/MAINTAINERS b/MAINTAINERS
index 1b79767d61..448dc951c5 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2164,6 +2164,7 @@ F: hw/vfio/*
 F: include/hw/vfio/
 F: docs/igd-assign.txt
 F: docs/devel/migration/vfio.rst
+F: qapi/vfio.json
 
 vfio-ccw
 M: Eric Farman <farman@linux.ibm.com>
diff --git a/qapi/qapi-schema.json b/qapi/qapi-schema.json
index 5e33da7228..b1581988e4 100644
--- a/qapi/qapi-schema.json
+++ b/qapi/qapi-schema.json
@@ -78,5 +78,6 @@
 { 'include': 'pci.json' }
 { 'include': 'stats.json' }
 { 'include': 'virtio.json' }
+{ 'include': 'vfio.json' }
 { 'include': 'cryptodev.json' }
 { 'include': 'cxl.json' }
diff --git a/qapi/vfio.json b/qapi/vfio.json
new file mode 100644
index 0000000000..a0e5013188
--- /dev/null
+++ b/qapi/vfio.json
@@ -0,0 +1,67 @@
+# -*- Mode: Python -*-
+# vim: filetype=python
+#
+
+##
+# = VFIO devices
+##
+
+##
+# @VfioMigrationState:
+#
+# An enumeration of the VFIO device migration states.
+#
+# @stop: The device is stopped.
+#
+# @running: The device is running.
+#
+# @stop-copy: The device is stopped and its internal state is available
+#     for reading.
+#
+# @resuming: The device is stopped and its internal state is available
+#     for writing.
+#
+# @running-p2p: The device is running in the P2P quiescent state.
+#
+# @pre-copy: The device is running, tracking its internal state and its
+#     internal state is available for reading.
+#
+# @pre-copy-p2p: The device is running in the P2P quiescent state,
+#     tracking its internal state and its internal state is available
+#     for reading.
+#
+# Since: 9.1
+##
+{ 'enum': 'VfioMigrationState',
+  'data': [ 'stop', 'running', 'stop-copy', 'resuming', 'running-p2p',
+            'pre-copy', 'pre-copy-p2p' ],
+  'prefix': 'QAPI_VFIO_MIGRATION_STATE' }
+
+##
+# @VFIO_MIGRATION:
+#
+# This event is emitted when a VFIO device migration state is changed.
+#
+# @device-id: The device's id, if it has one.
+#
+# @qom-path: The device's QOM path.
+#
+# @device-state: The new changed device migration state.
+#
+# Since: 9.1
+#
+# Example:
+#
+#     <- { "timestamp": { "seconds": 1713771323, "microseconds": 212268 },
+#          "event": "VFIO_MIGRATION",
+#          "data": {
+#              "device-id": "vfio_dev1",
+#              "qom-path": "/machine/peripheral/vfio_dev1",
+#              "device-state": "stop" } }
+##
+{ 'event': 'VFIO_MIGRATION',
+  'data': {
+      'device-id': 'str',
+      'qom-path': 'str',
+      'device-state': 'VfioMigrationState'
+  } }
diff --git a/qapi/meson.build b/qapi/meson.build
index c92af6e063..e7bc54e5d0 100644
--- a/qapi/meson.build
+++ b/qapi/meson.build
@@ -52,6 +52,7 @@ qapi_all_modules = [
   'stats',
   'trace',
   'transaction',
+  'vfio',
   'virtio',
   'yank',
 ]
-- 
2.26.3

[PATCH v3 2/4] vfio/migration: Emit VFIO migration QAPI event

[Avihai Horon]

Emit VFIO migration QAPI event when a VFIO device changes its migration
state. This can be used by management applications to get updates on the
current state of the VFIO device for their own purposes.

A new per VFIO device capability, "migration-events", is added so events
can be enabled only for the required devices. It is disabled by default.

Signed-off-by: Avihai Horon <avihaih@nvidia.com>
---
 include/hw/vfio/vfio-common.h |  1 +
 hw/vfio/migration.c           | 59 +++++++++++++++++++++++++++++++++--
 hw/vfio/pci.c                 |  2 ++
 3 files changed, 59 insertions(+), 3 deletions(-)

diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h
index b9da6c08ef..3ec5f2425e 100644
--- a/include/hw/vfio/vfio-common.h
+++ b/include/hw/vfio/vfio-common.h
@@ -115,6 +115,7 @@ typedef struct VFIODevice {
     bool no_mmap;
     bool ram_block_discard_allowed;
     OnOffAuto enable_migration;
+    bool migration_events;
     VFIODeviceOps *ops;
     unsigned int num_irqs;
     unsigned int num_regions;
diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c
index 06ae40969b..2e1a8f6031 100644
--- a/hw/vfio/migration.c
+++ b/hw/vfio/migration.c
@@ -24,6 +24,7 @@
 #include "migration/register.h"
 #include "migration/blocker.h"
 #include "qapi/error.h"
+#include "qapi/qapi-events-vfio.h"
 #include "exec/ramlist.h"
 #include "exec/ram_addr.h"
 #include "pci.h"
@@ -80,6 +81,58 @@ static const char *mig_state_to_str(enum vfio_device_mig_state state)
     }
 }
 
+static VfioMigrationState
+mig_state_to_qapi_state(enum vfio_device_mig_state state)
+{
+    switch (state) {
+    case VFIO_DEVICE_STATE_STOP:
+        return QAPI_VFIO_MIGRATION_STATE_STOP;
+    case VFIO_DEVICE_STATE_RUNNING:
+        return QAPI_VFIO_MIGRATION_STATE_RUNNING;
+    case VFIO_DEVICE_STATE_STOP_COPY:
+        return QAPI_VFIO_MIGRATION_STATE_STOP_COPY;
+    case VFIO_DEVICE_STATE_RESUMING:
+        return QAPI_VFIO_MIGRATION_STATE_RESUMING;
+    case VFIO_DEVICE_STATE_RUNNING_P2P:
+        return QAPI_VFIO_MIGRATION_STATE_RUNNING_P2P;
+    case VFIO_DEVICE_STATE_PRE_COPY:
+        return QAPI_VFIO_MIGRATION_STATE_PRE_COPY;
+    case VFIO_DEVICE_STATE_PRE_COPY_P2P:
+        return QAPI_VFIO_MIGRATION_STATE_PRE_COPY_P2P;
+    default:
+        g_assert_not_reached();
+    }
+}
+
+static void vfio_migration_send_event(VFIODevice *vbasedev)
+{
+    VFIOMigration *migration = vbasedev->migration;
+    DeviceState *dev = vbasedev->dev;
+    g_autofree char *qom_path = NULL;
+    Object *obj;
+
+    if (!vbasedev->migration_events) {
+        return;
+    }
+
+    g_assert(vbasedev->ops->vfio_get_object);
+    obj = vbasedev->ops->vfio_get_object(vbasedev);
+    g_assert(obj);
+    qom_path = object_get_canonical_path(obj);
+
+    qapi_event_send_vfio_migration(
+        dev->id, qom_path, mig_state_to_qapi_state(migration->device_state));
+}
+
+static void vfio_migration_set_device_state(VFIODevice *vbasedev,
+                                            enum vfio_device_mig_state state)
+{
+    VFIOMigration *migration = vbasedev->migration;
+
+    migration->device_state = state;
+    vfio_migration_send_event(vbasedev);
+}
+
 static int vfio_migration_set_state(VFIODevice *vbasedev,
                                     enum vfio_device_mig_state new_state,
                                     enum vfio_device_mig_state recover_state)
@@ -125,12 +178,12 @@ static int vfio_migration_set_state(VFIODevice *vbasedev,
             goto reset_device;
         }
 
-        migration->device_state = recover_state;
+        vfio_migration_set_device_state(vbasedev, recover_state);
 
         return ret;
     }
 
-    migration->device_state = new_state;
+    vfio_migration_set_device_state(vbasedev, new_state);
     if (mig_state->data_fd != -1) {
         if (migration->data_fd != -1) {
             /*
@@ -156,7 +209,7 @@ reset_device:
                  strerror(errno));
     }
 
-    migration->device_state = VFIO_DEVICE_STATE_RUNNING;
+    vfio_migration_set_device_state(vbasedev, VFIO_DEVICE_STATE_RUNNING);
 
     return ret;
 }
diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
index 64780d1b79..8840602c50 100644
--- a/hw/vfio/pci.c
+++ b/hw/vfio/pci.c
@@ -3362,6 +3362,8 @@ static Property vfio_pci_dev_properties[] = {
                     VFIO_FEATURE_ENABLE_IGD_OPREGION_BIT, false),
     DEFINE_PROP_ON_OFF_AUTO("enable-migration", VFIOPCIDevice,
                             vbasedev.enable_migration, ON_OFF_AUTO_AUTO),
+    DEFINE_PROP_BOOL("migration-events", VFIOPCIDevice,
+                     vbasedev.migration_events, false),
     DEFINE_PROP_BOOL("x-no-mmap", VFIOPCIDevice, vbasedev.no_mmap, false),
     DEFINE_PROP_BOOL("x-balloon-allowed", VFIOPCIDevice,
                      vbasedev.ram_block_discard_allowed, false),
-- 
2.26.3

[PATCH v3 3/4] vfio/migration: Don't emit STOP_COPY VFIO migration QAPI event twice

[Avihai Horon]

When migrating a VFIO device that supports pre-copy, it is transitioned
to STOP_COPY twice: once in vfio_vmstate_change() and second time in
vfio_save_complete_precopy().

The second transition is harmless, as it's a STOP_COPY->STOP_COPY no-op
transition. However, with the newly added VFIO migration QAPI event, the
STOP_COPY event is undesirably emitted twice.

Prevent this by returning early in vfio_migration_set_state() if
new_state is the same as current device state.

Note that the STOP_COPY transition in vfio_save_complete_precopy() is
essential for VFIO devices that don't support pre-copy, for migrating an
already stopped guest and for snapshots.

Signed-off-by: Avihai Horon <avihaih@nvidia.com>
Reviewed-by: Cédric Le Goater <clg@redhat.com>
---
 hw/vfio/migration.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c
index 2e1a8f6031..f2b7a3067b 100644
--- a/hw/vfio/migration.c
+++ b/hw/vfio/migration.c
@@ -146,6 +146,10 @@ static int vfio_migration_set_state(VFIODevice *vbasedev,
         (struct vfio_device_feature_mig_state *)feature->data;
     int ret;
 
+    if (new_state == migration->device_state) {
+        return 0;
+    }
+
     feature->argsz = sizeof(buf);
     feature->flags =
         VFIO_DEVICE_FEATURE_SET | VFIO_DEVICE_FEATURE_MIG_DEVICE_STATE;
-- 
2.26.3

[PATCH v3 4/4] vfio/migration: Enhance VFIO migration state tracing

[Avihai Horon]

Move trace_vfio_migration_set_state() to the top of the function, add
recover_state to it, and add a new trace event to
vfio_migration_set_device_state().

This improves tracing of device state changes as state changes are now
also logged when vfio_migration_set_state() fails (covering recover
state and device reset transitions) and in no-op state transitions to
the same state.

Suggested-by: Cédric Le Goater <clg@redhat.com>
Signed-off-by: Avihai Horon <avihaih@nvidia.com>
---
 hw/vfio/migration.c  | 8 ++++++--
 hw/vfio/trace-events | 3 ++-
 2 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c
index f2b7a3067b..7f0d76ab50 100644
--- a/hw/vfio/migration.c
+++ b/hw/vfio/migration.c
@@ -129,6 +129,9 @@ static void vfio_migration_set_device_state(VFIODevice *vbasedev,
 {
     VFIOMigration *migration = vbasedev->migration;
 
+    trace_vfio_migration_set_device_state(vbasedev->name,
+                                          mig_state_to_str(state));
+
     migration->device_state = state;
     vfio_migration_send_event(vbasedev);
 }
@@ -146,6 +149,9 @@ static int vfio_migration_set_state(VFIODevice *vbasedev,
         (struct vfio_device_feature_mig_state *)feature->data;
     int ret;
 
+    trace_vfio_migration_set_state(vbasedev->name, mig_state_to_str(new_state),
+                                   mig_state_to_str(recover_state));
+
     if (new_state == migration->device_state) {
         return 0;
     }
@@ -203,8 +209,6 @@ static int vfio_migration_set_state(VFIODevice *vbasedev,
         migration->data_fd = mig_state->data_fd;
     }
 
-    trace_vfio_migration_set_state(vbasedev->name, mig_state_to_str(new_state));
-
     return 0;
 
 reset_device:
diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events
index f0474b244b..64161bf6f4 100644
--- a/hw/vfio/trace-events
+++ b/hw/vfio/trace-events
@@ -152,7 +152,8 @@ vfio_load_device_config_state(const char *name) " (%s)"
 vfio_load_state(const char *name, uint64_t data) " (%s) data 0x%"PRIx64
 vfio_load_state_device_data(const char *name, uint64_t data_size, int ret) " (%s) size 0x%"PRIx64" ret %d"
 vfio_migration_realize(const char *name) " (%s)"
-vfio_migration_set_state(const char *name, const char *state) " (%s) state %s"
+vfio_migration_set_device_state(const char *name, const char *state) " (%s) state %s"
+vfio_migration_set_state(const char *name, const char *new_state, const char *recover_state) " (%s) new state %s, recover state %s"
 vfio_migration_state_notifier(const char *name, int state) " (%s) state %d"
 vfio_save_block(const char *name, int data_size) " (%s) data_size %d"
 vfio_save_cleanup(const char *name) " (%s)"
-- 
2.26.3

Re: [PATCH v3 4/4] vfio/migration: Enhance VFIO migration state tracing

[Cédric Le Goater]

On 5/15/24 15:21, Avihai Horon wrote:
> Move trace_vfio_migration_set_state() to the top of the function, add
> recover_state to it, and add a new trace event to
> vfio_migration_set_device_state().
> 
> This improves tracing of device state changes as state changes are now
> also logged when vfio_migration_set_state() fails (covering recover
> state and device reset transitions) and in no-op state transitions to
> the same state.
> 
> Suggested-by: Cédric Le Goater <clg@redhat.com>
> Signed-off-by: Avihai Horon <avihaih@nvidia.com>


Thanks for doing so,



Reviewed-by: Cédric Le Goater <clg@redhat.com>


C.


> ---
>   hw/vfio/migration.c  | 8 ++++++--
>   hw/vfio/trace-events | 3 ++-
>   2 files changed, 8 insertions(+), 3 deletions(-)
> 
> diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c
> index f2b7a3067b..7f0d76ab50 100644
> --- a/hw/vfio/migration.c
> +++ b/hw/vfio/migration.c
> @@ -129,6 +129,9 @@ static void vfio_migration_set_device_state(VFIODevice *vbasedev,
>   {
>       VFIOMigration *migration = vbasedev->migration;
>   
> +    trace_vfio_migration_set_device_state(vbasedev->name,
> +                                          mig_state_to_str(state));
> +
>       migration->device_state = state;
>       vfio_migration_send_event(vbasedev);
>   }
> @@ -146,6 +149,9 @@ static int vfio_migration_set_state(VFIODevice *vbasedev,
>           (struct vfio_device_feature_mig_state *)feature->data;
>       int ret;
>   
> +    trace_vfio_migration_set_state(vbasedev->name, mig_state_to_str(new_state),
> +                                   mig_state_to_str(recover_state));
> +
>       if (new_state == migration->device_state) {
>           return 0;
>       }
> @@ -203,8 +209,6 @@ static int vfio_migration_set_state(VFIODevice *vbasedev,
>           migration->data_fd = mig_state->data_fd;
>       }
>   
> -    trace_vfio_migration_set_state(vbasedev->name, mig_state_to_str(new_state));
> -
>       return 0;
>   
>   reset_device:
> diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events
> index f0474b244b..64161bf6f4 100644
> --- a/hw/vfio/trace-events
> +++ b/hw/vfio/trace-events
> @@ -152,7 +152,8 @@ vfio_load_device_config_state(const char *name) " (%s)"
>   vfio_load_state(const char *name, uint64_t data) " (%s) data 0x%"PRIx64
>   vfio_load_state_device_data(const char *name, uint64_t data_size, int ret) " (%s) size 0x%"PRIx64" ret %d"
>   vfio_migration_realize(const char *name) " (%s)"
> -vfio_migration_set_state(const char *name, const char *state) " (%s) state %s"
> +vfio_migration_set_device_state(const char *name, const char *state) " (%s) state %s"
> +vfio_migration_set_state(const char *name, const char *new_state, const char *recover_state) " (%s) new state %s, recover state %s"
>   vfio_migration_state_notifier(const char *name, int state) " (%s) state %d"
>   vfio_save_block(const char *name, int data_size) " (%s) data_size %d"
>   vfio_save_cleanup(const char *name) " (%s)"

Re: [PATCH v3 2/4] vfio/migration: Emit VFIO migration QAPI event

[Cédric Le Goater]

On 5/15/24 15:21, Avihai Horon wrote:
> Emit VFIO migration QAPI event when a VFIO device changes its migration
> state. This can be used by management applications to get updates on the
> current state of the VFIO device for their own purposes.
> 
> A new per VFIO device capability, "migration-events", is added so events
> can be enabled only for the required devices. It is disabled by default.
> 
> Signed-off-by: Avihai Horon <avihaih@nvidia.com>


Reviewed-by: Cédric Le Goater <clg@redhat.com>

Thanks,

C.


> ---
>   include/hw/vfio/vfio-common.h |  1 +
>   hw/vfio/migration.c           | 59 +++++++++++++++++++++++++++++++++--
>   hw/vfio/pci.c                 |  2 ++
>   3 files changed, 59 insertions(+), 3 deletions(-)
> 
> diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h
> index b9da6c08ef..3ec5f2425e 100644
> --- a/include/hw/vfio/vfio-common.h
> +++ b/include/hw/vfio/vfio-common.h
> @@ -115,6 +115,7 @@ typedef struct VFIODevice {
>       bool no_mmap;
>       bool ram_block_discard_allowed;
>       OnOffAuto enable_migration;
> +    bool migration_events;
>       VFIODeviceOps *ops;
>       unsigned int num_irqs;
>       unsigned int num_regions;
> diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c
> index 06ae40969b..2e1a8f6031 100644
> --- a/hw/vfio/migration.c
> +++ b/hw/vfio/migration.c
> @@ -24,6 +24,7 @@
>   #include "migration/register.h"
>   #include "migration/blocker.h"
>   #include "qapi/error.h"
> +#include "qapi/qapi-events-vfio.h"
>   #include "exec/ramlist.h"
>   #include "exec/ram_addr.h"
>   #include "pci.h"
> @@ -80,6 +81,58 @@ static const char *mig_state_to_str(enum vfio_device_mig_state state)
>       }
>   }
>   
> +static VfioMigrationState
> +mig_state_to_qapi_state(enum vfio_device_mig_state state)
> +{
> +    switch (state) {
> +    case VFIO_DEVICE_STATE_STOP:
> +        return QAPI_VFIO_MIGRATION_STATE_STOP;
> +    case VFIO_DEVICE_STATE_RUNNING:
> +        return QAPI_VFIO_MIGRATION_STATE_RUNNING;
> +    case VFIO_DEVICE_STATE_STOP_COPY:
> +        return QAPI_VFIO_MIGRATION_STATE_STOP_COPY;
> +    case VFIO_DEVICE_STATE_RESUMING:
> +        return QAPI_VFIO_MIGRATION_STATE_RESUMING;
> +    case VFIO_DEVICE_STATE_RUNNING_P2P:
> +        return QAPI_VFIO_MIGRATION_STATE_RUNNING_P2P;
> +    case VFIO_DEVICE_STATE_PRE_COPY:
> +        return QAPI_VFIO_MIGRATION_STATE_PRE_COPY;
> +    case VFIO_DEVICE_STATE_PRE_COPY_P2P:
> +        return QAPI_VFIO_MIGRATION_STATE_PRE_COPY_P2P;
> +    default:
> +        g_assert_not_reached();
> +    }
> +}
> +
> +static void vfio_migration_send_event(VFIODevice *vbasedev)
> +{
> +    VFIOMigration *migration = vbasedev->migration;
> +    DeviceState *dev = vbasedev->dev;
> +    g_autofree char *qom_path = NULL;
> +    Object *obj;
> +
> +    if (!vbasedev->migration_events) {
> +        return;
> +    }
> +
> +    g_assert(vbasedev->ops->vfio_get_object);
> +    obj = vbasedev->ops->vfio_get_object(vbasedev);
> +    g_assert(obj);
> +    qom_path = object_get_canonical_path(obj);
> +
> +    qapi_event_send_vfio_migration(
> +        dev->id, qom_path, mig_state_to_qapi_state(migration->device_state));
> +}
> +
> +static void vfio_migration_set_device_state(VFIODevice *vbasedev,
> +                                            enum vfio_device_mig_state state)
> +{
> +    VFIOMigration *migration = vbasedev->migration;
> +
> +    migration->device_state = state;
> +    vfio_migration_send_event(vbasedev);
> +}
> +
>   static int vfio_migration_set_state(VFIODevice *vbasedev,
>                                       enum vfio_device_mig_state new_state,
>                                       enum vfio_device_mig_state recover_state)
> @@ -125,12 +178,12 @@ static int vfio_migration_set_state(VFIODevice *vbasedev,
>               goto reset_device;
>           }
>   
> -        migration->device_state = recover_state;
> +        vfio_migration_set_device_state(vbasedev, recover_state);
>   
>           return ret;
>       }
>   
> -    migration->device_state = new_state;
> +    vfio_migration_set_device_state(vbasedev, new_state);
>       if (mig_state->data_fd != -1) {
>           if (migration->data_fd != -1) {
>               /*
> @@ -156,7 +209,7 @@ reset_device:
>                    strerror(errno));
>       }
>   
> -    migration->device_state = VFIO_DEVICE_STATE_RUNNING;
> +    vfio_migration_set_device_state(vbasedev, VFIO_DEVICE_STATE_RUNNING);
>   
>       return ret;
>   }
> diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
> index 64780d1b79..8840602c50 100644
> --- a/hw/vfio/pci.c
> +++ b/hw/vfio/pci.c
> @@ -3362,6 +3362,8 @@ static Property vfio_pci_dev_properties[] = {
>                       VFIO_FEATURE_ENABLE_IGD_OPREGION_BIT, false),
>       DEFINE_PROP_ON_OFF_AUTO("enable-migration", VFIOPCIDevice,
>                               vbasedev.enable_migration, ON_OFF_AUTO_AUTO),
> +    DEFINE_PROP_BOOL("migration-events", VFIOPCIDevice,
> +                     vbasedev.migration_events, false),
>       DEFINE_PROP_BOOL("x-no-mmap", VFIOPCIDevice, vbasedev.no_mmap, false),
>       DEFINE_PROP_BOOL("x-balloon-allowed", VFIOPCIDevice,
>                        vbasedev.ram_block_discard_allowed, false),

Re: [PATCH v3 0/4] qapi/vfio: Add VFIO migration QAPI event

[Cédric Le Goater]

On 5/15/24 15:21, Avihai Horon wrote:
> Hello,
> 
> This series adds a new QAPI event for VFIO device migration state
> change. This event will be emitted when a VFIO device changes its
> state, for example, during migration or when stopping/starting the
> guest.
> 
> This event can be used by management applications to get updates on the
> current state of the VFIO device for their own purposes.
> 
> A new per VFIO device capability, "migration-events", is added so events
> can be enabled only for the required devices. It is disabled by default.
> 
> Feedback/comments are appreciated,


Applied to vfio-next.

Thanks,

C.


> 
> Thanks.
> 
> Changes from v2 [2]:
> * Added assert for vbasedev->ops->vfio_get_object and obj. (Cedric)
> * Renamed set_state() to vfio_migration_set_device_state(). (Cedric)
> * Enhanced tracing of device state change. (Cedric)
> * Added Cedric's R-b.
> 
> Changes from v1 [1]:
> * Added more info to patch #1 commit mesasge. (Markus)
> * Renamed VFIODeviceMigState to VfioMigrationState and
>    VFIO_DEVICE_MIG_STATE_CHANGED to VFIO_MIGRATION. (Joao, Markus)
> * Added qom-path and qdev id to VFIO_MIGRATION event data. (Markus)
> * Handled no-op state transitions in vfio_migration_set_state().
>    (Cedric)
> * Added helper to set VFIO state and emit VFIO event. (Peter)
> 
> [1]
> https://lore.kernel.org/qemu-devel/20240430051621.19597-1-avihaih@nvidia.com/
> 
> [2]
> https://lore.kernel.org/qemu-devel/20240509090954.16447-1-avihaih@nvidia.com/
> 
> Avihai Horon (4):
>    qapi/vfio: Add VFIO migration QAPI event
>    vfio/migration: Emit VFIO migration QAPI event
>    vfio/migration: Don't emit STOP_COPY VFIO migration QAPI event twice
>    vfio/migration: Enhance VFIO migration state tracing
> 
>   MAINTAINERS                   |  1 +
>   qapi/qapi-schema.json         |  1 +
>   qapi/vfio.json                | 67 +++++++++++++++++++++++++++++++++
>   include/hw/vfio/vfio-common.h |  1 +
>   hw/vfio/migration.c           | 71 ++++++++++++++++++++++++++++++++---
>   hw/vfio/pci.c                 |  2 +
>   hw/vfio/trace-events          |  3 +-
>   qapi/meson.build              |  1 +
>   8 files changed, 141 insertions(+), 6 deletions(-)
>   create mode 100644 qapi/vfio.json
>