[PATCH v7 0/5] vEVENTQ support for accelerated SMMUv3 devices

[PATCH v7 0/5] vEVENTQ support for accelerated SMMUv3 devices

[Shameer Kolothum]

Hi,

Changes from v6:
 https://lore.kernel.org/qemu-devel/20260213103942.142823-1-skolothumtho@nvidia.com/
 
 -Addressed feedback and picked up R-by/T-by tags. Thanks!
 -Updated error report (patch #3)

Please take a look and let me know.

Thanks,
Shameer

---
Background(From v2):

When accel=on is enabled for an SMMUv3 instance, the host hardware SMMUv3
may generate Stage-1 (S1) event notifications that are intended for the
vIOMMU instance in userspace.

This series adds QEMU support for receiving such host events through the
vEVENTQ interface and propagating them to the guest. The implementation
uses the vEVENTQ support provided by the IOMMUFD subsystem in the kernel.

I have tested this on a Grace platform using some hacks to generate
events. Further testing and feedback are welcome.
---


Nicolin Chen (2):
  backends/iommufd: Introduce iommufd_backend_alloc_veventq
  hw/arm/smmuv3-accel: Allocate vEVENTQ for accelerated SMMUv3 devices

Shameer Kolothum (3):
  hw/arm/smmuv3-accel: Add viommu free helper
  hw/arm/smmuv3: Introduce a helper function for event propagation
  hw/arm/smmuv3-accel: Read and propagate host vIOMMU events

 hw/arm/smmuv3-accel.h    |   6 ++
 hw/arm/smmuv3-internal.h |   4 ++
 include/system/iommufd.h |  14 ++++
 backends/iommufd.c       |  31 +++++++++
 hw/arm/smmuv3-accel.c    | 146 ++++++++++++++++++++++++++++++++++++---
 hw/arm/smmuv3.c          |  26 +++++--
 backends/trace-events    |   1 +
 hw/arm/trace-events      |   2 +-
 8 files changed, 215 insertions(+), 15 deletions(-)

-- 
2.43.0

[PATCH v7 1/5] backends/iommufd: Introduce iommufd_backend_alloc_veventq

[Shameer Kolothum]

From: Nicolin Chen <nicolinc@nvidia.com>

Add a new helper for IOMMU_VEVENTQ_ALLOC ioctl to allocate a virtual event
queue (vEVENTQ) for a vIOMMU object, and define struct IOMMUFDVeventq to
represent the allocated vEVENTQ.

Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Shameer Kolothum <skolothumtho@nvidia.com>
---
 include/system/iommufd.h | 14 ++++++++++++++
 backends/iommufd.c       | 31 +++++++++++++++++++++++++++++++
 backends/trace-events    |  1 +
 3 files changed, 46 insertions(+)

diff --git a/include/system/iommufd.h b/include/system/iommufd.h
index 80d72469a9..7062944fe6 100644
--- a/include/system/iommufd.h
+++ b/include/system/iommufd.h
@@ -56,6 +56,15 @@ typedef struct IOMMUFDVdev {
     uint32_t virt_id;  /* virtual device ID */
 } IOMMUFDVdev;
 
+/* Virtual event queue interface for a vIOMMU */
+typedef struct IOMMUFDVeventq {
+    IOMMUFDViommu *viommu;
+    uint32_t veventq_id;
+    uint32_t veventq_fd;
+    uint32_t last_event_seq; /* Sequence number of last processed event */
+    bool event_start; /* True after first valid event; cleared on overflow */
+} IOMMUFDVeventq;
+
 bool iommufd_backend_connect(IOMMUFDBackend *be, Error **errp);
 void iommufd_backend_disconnect(IOMMUFDBackend *be);
 
@@ -86,6 +95,11 @@ bool iommufd_backend_alloc_vdev(IOMMUFDBackend *be, uint32_t dev_id,
                                 uint32_t viommu_id, uint64_t virt_id,
                                 uint32_t *out_vdev_id, Error **errp);
 
+bool iommufd_backend_alloc_veventq(IOMMUFDBackend *be, uint32_t viommu_id,
+                                   uint32_t type, uint32_t depth,
+                                   uint32_t *out_veventq_id,
+                                   uint32_t *out_veventq_fd, Error **errp);
+
 bool iommufd_backend_set_dirty_tracking(IOMMUFDBackend *be, uint32_t hwpt_id,
                                         bool start, Error **errp);
 bool iommufd_backend_get_dirty_bitmap(IOMMUFDBackend *be, uint32_t hwpt_id,
diff --git a/backends/iommufd.c b/backends/iommufd.c
index 13822df82f..acfab907c0 100644
--- a/backends/iommufd.c
+++ b/backends/iommufd.c
@@ -504,6 +504,37 @@ bool iommufd_backend_alloc_vdev(IOMMUFDBackend *be, uint32_t dev_id,
     return true;
 }
 
+bool iommufd_backend_alloc_veventq(IOMMUFDBackend *be, uint32_t viommu_id,
+                                   uint32_t type, uint32_t depth,
+                                   uint32_t *out_veventq_id,
+                                   uint32_t *out_veventq_fd, Error **errp)
+{
+    int ret;
+    struct iommu_veventq_alloc alloc_veventq = {
+        .size = sizeof(alloc_veventq),
+        .flags = 0,
+        .type = type,
+        .veventq_depth = depth,
+        .viommu_id = viommu_id,
+    };
+
+    ret = ioctl(be->fd, IOMMU_VEVENTQ_ALLOC, &alloc_veventq);
+
+    trace_iommufd_viommu_alloc_eventq(be->fd, viommu_id, type,
+                                      alloc_veventq.out_veventq_id,
+                                      alloc_veventq.out_veventq_fd, ret);
+    if (ret) {
+        error_setg_errno(errp, errno, "IOMMU_VEVENTQ_ALLOC failed");
+        return false;
+    }
+
+    g_assert(out_veventq_id);
+    g_assert(out_veventq_fd);
+    *out_veventq_id = alloc_veventq.out_veventq_id;
+    *out_veventq_fd = alloc_veventq.out_veventq_fd;
+    return true;
+}
+
 bool host_iommu_device_iommufd_attach_hwpt(HostIOMMUDeviceIOMMUFD *idev,
                                            uint32_t hwpt_id, Error **errp)
 {
diff --git a/backends/trace-events b/backends/trace-events
index 8dc64a20d3..b9365113e7 100644
--- a/backends/trace-events
+++ b/backends/trace-events
@@ -23,6 +23,7 @@ iommufd_backend_get_dirty_bitmap(int iommufd, uint32_t hwpt_id, uint64_t iova, u
 iommufd_backend_invalidate_cache(int iommufd, uint32_t id, uint32_t data_type, uint32_t entry_len, uint32_t entry_num, uint32_t done_num, uint64_t data_ptr, int ret) " iommufd=%d id=%u data_type=%u entry_len=%u entry_num=%u done_num=%u data_ptr=0x%"PRIx64" (%d)"
 iommufd_backend_alloc_viommu(int iommufd, uint32_t dev_id, uint32_t type, uint32_t hwpt_id, uint32_t viommu_id, int ret) " iommufd=%d type=%u dev_id=%u hwpt_id=%u viommu_id=%u (%d)"
 iommufd_backend_alloc_vdev(int iommufd, uint32_t dev_id, uint32_t viommu_id, uint64_t virt_id, uint32_t vdev_id, int ret) " iommufd=%d dev_id=%u viommu_id=%u virt_id=0x%"PRIx64" vdev_id=%u (%d)"
+iommufd_viommu_alloc_eventq(int iommufd, uint32_t viommu_id, uint32_t type, uint32_t veventq_id, uint32_t veventq_fd, int ret) " iommufd=%d viommu_id=%u type=%u veventq_id=%u veventq_fd=%u (%d)"
 
 # igvm-cfg.c
 igvm_reset_enter(int type) "type=%u"
-- 
2.43.0

[PATCH v7 2/5] hw/arm/smmuv3-accel: Add viommu free helper

[Shameer Kolothum]

Move viommu teardown into a helper function and use it from the
last device removal path. No functional change.

Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Shameer Kolothum <skolothumtho@nvidia.com>
---
 hw/arm/smmuv3-accel.c | 21 +++++++++++++++------
 1 file changed, 15 insertions(+), 6 deletions(-)

diff --git a/hw/arm/smmuv3-accel.c b/hw/arm/smmuv3-accel.c
index f5cd4df336..c19c526fca 100644
--- a/hw/arm/smmuv3-accel.c
+++ b/hw/arm/smmuv3-accel.c
@@ -390,6 +390,20 @@ bool smmuv3_accel_issue_inv_cmd(SMMUv3State *bs, void *cmd, SMMUDevice *sdev,
                    sizeof(Cmd), &entry_num, cmd, errp);
 }
 
+static void smmuv3_accel_free_viommu(SMMUv3AccelState *accel)
+{
+    IOMMUFDViommu *viommu = accel->viommu;
+
+    if (!viommu) {
+        return;
+    }
+    iommufd_backend_free_id(viommu->iommufd, accel->bypass_hwpt_id);
+    iommufd_backend_free_id(viommu->iommufd, accel->abort_hwpt_id);
+    iommufd_backend_free_id(viommu->iommufd, accel->viommu->viommu_id);
+    g_free(viommu);
+    accel->viommu = NULL;
+}
+
 static bool
 smmuv3_accel_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
                           Error **errp)
@@ -549,12 +563,7 @@ static void smmuv3_accel_unset_iommu_device(PCIBus *bus, void *opaque,
     trace_smmuv3_accel_unset_iommu_device(devfn, idev->devid);
 
     if (QLIST_EMPTY(&accel->device_list)) {
-        iommufd_backend_free_id(accel->viommu->iommufd, accel->bypass_hwpt_id);
-        iommufd_backend_free_id(accel->viommu->iommufd, accel->abort_hwpt_id);
-        iommufd_backend_free_id(accel->viommu->iommufd,
-                                accel->viommu->viommu_id);
-        g_free(accel->viommu);
-        accel->viommu = NULL;
+        smmuv3_accel_free_viommu(accel);
     }
 }
 
-- 
2.43.0

[PATCH v7 3/5] hw/arm/smmuv3-accel: Allocate vEVENTQ for accelerated SMMUv3 devices

[Shameer Kolothum]

From: Nicolin Chen <nicolinc@nvidia.com>

When the guest enables the Event Queue and a vIOMMU is present, allocate a
vEVENTQ object so that host-side events related to the vIOMMU can be
received and propagated back to the guest.

For cold-plugged devices using SMMUv3 acceleration, the vIOMMU is created
before the guest boots. In this case, the vEVENTQ is allocated when the
guest writes to SMMU_CR0 and sets EVENTQEN = 1.

If no cold-plugged device exists at boot (i.e. no vIOMMU initially), the
vEVENTQ is allocated when a vIOMMU is created, i.e. during the first
device hot-plug.

Errors from command queue consumption and vEVENTQ allocation are reported
independently as the two operations are unrelated.

Event read and propagation will be added in a later patch.

Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Shameer Kolothum <skolothumtho@nvidia.com>
---
 hw/arm/smmuv3-accel.h |  6 +++++
 hw/arm/smmuv3-accel.c | 61 +++++++++++++++++++++++++++++++++++++++++--
 hw/arm/smmuv3.c       |  6 +++++
 3 files changed, 71 insertions(+), 2 deletions(-)

diff --git a/hw/arm/smmuv3-accel.h b/hw/arm/smmuv3-accel.h
index a8a64802ec..dba6c71de5 100644
--- a/hw/arm/smmuv3-accel.h
+++ b/hw/arm/smmuv3-accel.h
@@ -22,6 +22,7 @@
  */
 typedef struct SMMUv3AccelState {
     IOMMUFDViommu *viommu;
+    IOMMUFDVeventq *veventq;
     uint32_t bypass_hwpt_id;
     uint32_t abort_hwpt_id;
     QLIST_HEAD(, SMMUv3AccelDevice) device_list;
@@ -50,6 +51,7 @@ bool smmuv3_accel_attach_gbpa_hwpt(SMMUv3State *s, Error **errp);
 bool smmuv3_accel_issue_inv_cmd(SMMUv3State *s, void *cmd, SMMUDevice *sdev,
                                 Error **errp);
 void smmuv3_accel_idr_override(SMMUv3State *s);
+bool smmuv3_accel_alloc_veventq(SMMUv3State *s, Error **errp);
 void smmuv3_accel_reset(SMMUv3State *s);
 #else
 static inline void smmuv3_accel_init(SMMUv3State *s)
@@ -80,6 +82,10 @@ smmuv3_accel_issue_inv_cmd(SMMUv3State *s, void *cmd, SMMUDevice *sdev,
 static inline void smmuv3_accel_idr_override(SMMUv3State *s)
 {
 }
+static inline bool smmuv3_accel_alloc_veventq(SMMUv3State *s, Error **errp)
+{
+    return true;
+}
 static inline void smmuv3_accel_reset(SMMUv3State *s)
 {
 }
diff --git a/hw/arm/smmuv3-accel.c b/hw/arm/smmuv3-accel.c
index c19c526fca..f703ea1aac 100644
--- a/hw/arm/smmuv3-accel.c
+++ b/hw/arm/smmuv3-accel.c
@@ -390,6 +390,19 @@ bool smmuv3_accel_issue_inv_cmd(SMMUv3State *bs, void *cmd, SMMUDevice *sdev,
                    sizeof(Cmd), &entry_num, cmd, errp);
 }
 
+static void smmuv3_accel_free_veventq(SMMUv3AccelState *accel)
+{
+    IOMMUFDVeventq *veventq = accel->veventq;
+
+    if (!veventq) {
+        return;
+    }
+    close(veventq->veventq_fd);
+    iommufd_backend_free_id(accel->viommu->iommufd, veventq->veventq_id);
+    g_free(veventq);
+    accel->veventq = NULL;
+}
+
 static void smmuv3_accel_free_viommu(SMMUv3AccelState *accel)
 {
     IOMMUFDViommu *viommu = accel->viommu;
@@ -397,6 +410,7 @@ static void smmuv3_accel_free_viommu(SMMUv3AccelState *accel)
     if (!viommu) {
         return;
     }
+    smmuv3_accel_free_veventq(accel);
     iommufd_backend_free_id(viommu->iommufd, accel->bypass_hwpt_id);
     iommufd_backend_free_id(viommu->iommufd, accel->abort_hwpt_id);
     iommufd_backend_free_id(viommu->iommufd, accel->viommu->viommu_id);
@@ -404,6 +418,41 @@ static void smmuv3_accel_free_viommu(SMMUv3AccelState *accel)
     accel->viommu = NULL;
 }
 
+bool smmuv3_accel_alloc_veventq(SMMUv3State *s, Error **errp)
+{
+    SMMUv3AccelState *accel = s->s_accel;
+    IOMMUFDVeventq *veventq;
+    uint32_t veventq_id;
+    uint32_t veventq_fd;
+
+    if (!accel || !accel->viommu) {
+        return true;
+    }
+
+    if (accel->veventq) {
+        return true;
+    }
+
+    if (!smmuv3_eventq_enabled(s)) {
+        return true;
+    }
+
+    if (!iommufd_backend_alloc_veventq(accel->viommu->iommufd,
+                                       accel->viommu->viommu_id,
+                                       IOMMU_VEVENTQ_TYPE_ARM_SMMUV3,
+                                       1 << s->eventq.log2size, &veventq_id,
+                                       &veventq_fd, errp)) {
+        return false;
+    }
+
+    veventq = g_new0(IOMMUFDVeventq, 1);
+    veventq->veventq_id = veventq_id;
+    veventq->veventq_fd = veventq_fd;
+    veventq->viommu = accel->viommu;
+    accel->veventq = veventq;
+    return true;
+}
+
 static bool
 smmuv3_accel_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
                           Error **errp)
@@ -429,6 +478,7 @@ smmuv3_accel_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
     viommu->viommu_id = viommu_id;
     viommu->s2_hwpt_id = s2_hwpt_id;
     viommu->iommufd = idev->iommufd;
+    accel->viommu = viommu;
 
     /*
      * Pre-allocate HWPTs for S1 bypass and abort cases. These will be attached
@@ -448,14 +498,20 @@ smmuv3_accel_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
         goto free_abort_hwpt;
     }
 
+    /* Allocate a vEVENTQ if guest has enabled event queue */
+    if (!smmuv3_accel_alloc_veventq(s, errp)) {
+        goto free_bypass_hwpt;
+    }
+
     /* Attach a HWPT based on SMMUv3 GBPA.ABORT value */
     hwpt_id = smmuv3_accel_gbpa_hwpt(s, accel);
     if (!host_iommu_device_iommufd_attach_hwpt(idev, hwpt_id, errp)) {
-        goto free_bypass_hwpt;
+        goto free_veventq;
     }
-    accel->viommu = viommu;
     return true;
 
+free_veventq:
+    smmuv3_accel_free_veventq(accel);
 free_bypass_hwpt:
     iommufd_backend_free_id(idev->iommufd, accel->bypass_hwpt_id);
 free_abort_hwpt:
@@ -463,6 +519,7 @@ free_abort_hwpt:
 free_viommu:
     iommufd_backend_free_id(idev->iommufd, viommu->viommu_id);
     g_free(viommu);
+    accel->viommu = NULL;
     return false;
 }
 
diff --git a/hw/arm/smmuv3.c b/hw/arm/smmuv3.c
index c08d58c579..5d718da764 100644
--- a/hw/arm/smmuv3.c
+++ b/hw/arm/smmuv3.c
@@ -1605,6 +1605,12 @@ static MemTxResult smmu_writel(SMMUv3State *s, hwaddr offset,
         s->cr0ack = data & ~SMMU_CR0_RESERVED;
         /* in case the command queue has been enabled */
         smmuv3_cmdq_consume(s, &local_err);
+        if (local_err) {
+            error_report_err(local_err);
+            local_err = NULL;
+        }
+        /* Allocate vEVENTQ if EVENTQ is enabled and a vIOMMU is available */
+        smmuv3_accel_alloc_veventq(s, &local_err);
         break;
     case A_CR1:
         s->cr[1] = data;
-- 
2.43.0

[PATCH v7 4/5] hw/arm/smmuv3: Introduce a helper function for event propagation

[Shameer Kolothum]

Factor out the code that propagates event records to the guest into a
helper function. The accelerated SMMUv3 path can use this to propagate
host events in a subsequent patch.

Take the mutex inside the helper before accessing the Event Queue.
Today event propagation occurs only in the core SMMUv3 path and is
effectively serialized. A subsequent patch will also invoke this helper
from the accelerated event read path, which may run concurrently.
Therefore serialization is required here.

No functional change intended.

Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Shameer Kolothum <skolothumtho@nvidia.com>
---
 hw/arm/smmuv3-internal.h |  4 ++++
 hw/arm/smmuv3.c          | 20 ++++++++++++++------
 hw/arm/trace-events      |  2 +-
 3 files changed, 19 insertions(+), 7 deletions(-)

diff --git a/hw/arm/smmuv3-internal.h b/hw/arm/smmuv3-internal.h
index a6464425ec..b666109ad9 100644
--- a/hw/arm/smmuv3-internal.h
+++ b/hw/arm/smmuv3-internal.h
@@ -352,7 +352,11 @@ typedef struct SMMUEventInfo {
             (x)->word[6] = (uint32_t)(addr & 0xffffffff); \
     } while (0)
 
+#define EVT_GET_TYPE(x)  extract32((x)->word[0], 0, 8)
+#define EVT_GET_SID(x)   ((x)->word[1])
+
 void smmuv3_record_event(SMMUv3State *s, SMMUEventInfo *event);
+void smmuv3_propagate_event(SMMUv3State *s, Evt *evt);
 int smmu_find_ste(SMMUv3State *s, uint32_t sid, STE *ste, SMMUEventInfo *event);
 
 static inline int oas2bits(int oas_field)
diff --git a/hw/arm/smmuv3.c b/hw/arm/smmuv3.c
index 5d718da764..af7a54143f 100644
--- a/hw/arm/smmuv3.c
+++ b/hw/arm/smmuv3.c
@@ -168,10 +168,22 @@ static MemTxResult smmuv3_write_eventq(SMMUv3State *s, Evt *evt)
     return MEMTX_OK;
 }
 
+void smmuv3_propagate_event(SMMUv3State *s, Evt *evt)
+{
+    MemTxResult r;
+
+    trace_smmuv3_propagate_event(smmu_event_string(EVT_GET_TYPE(evt)),
+                                 EVT_GET_SID(evt));
+    QEMU_LOCK_GUARD(&s->mutex);
+    r = smmuv3_write_eventq(s, evt);
+    if (r != MEMTX_OK) {
+        smmuv3_trigger_irq(s, SMMU_IRQ_GERROR, R_GERROR_EVENTQ_ABT_ERR_MASK);
+    }
+}
+
 void smmuv3_record_event(SMMUv3State *s, SMMUEventInfo *info)
 {
     Evt evt = {};
-    MemTxResult r;
 
     if (!smmuv3_eventq_enabled(s)) {
         return;
@@ -251,11 +263,7 @@ void smmuv3_record_event(SMMUv3State *s, SMMUEventInfo *info)
         g_assert_not_reached();
     }
 
-    trace_smmuv3_record_event(smmu_event_string(info->type), info->sid);
-    r = smmuv3_write_eventq(s, &evt);
-    if (r != MEMTX_OK) {
-        smmuv3_trigger_irq(s, SMMU_IRQ_GERROR, R_GERROR_EVENTQ_ABT_ERR_MASK);
-    }
+    smmuv3_propagate_event(s, &evt);
     info->recorded = true;
 }
 
diff --git a/hw/arm/trace-events b/hw/arm/trace-events
index 8135c0c734..3457536fb0 100644
--- a/hw/arm/trace-events
+++ b/hw/arm/trace-events
@@ -40,7 +40,7 @@ smmuv3_cmdq_opcode(const char *opcode) "<--- %s"
 smmuv3_cmdq_consume_out(uint32_t prod, uint32_t cons, uint8_t prod_wrap, uint8_t cons_wrap) "prod:%d, cons:%d, prod_wrap:%d, cons_wrap:%d "
 smmuv3_cmdq_consume_error(const char *cmd_name, uint8_t cmd_error) "Error on %s command execution: %d"
 smmuv3_write_mmio(uint64_t addr, uint64_t val, unsigned size, uint32_t r) "addr: 0x%"PRIx64" val:0x%"PRIx64" size: 0x%x(%d)"
-smmuv3_record_event(const char *type, uint32_t sid) "%s sid=0x%x"
+smmuv3_propagate_event(const char *type, uint32_t sid) "%s sid=0x%x"
 smmuv3_find_ste(uint16_t sid, uint32_t features, uint16_t sid_split) "sid=0x%x features:0x%x, sid_split:0x%x"
 smmuv3_find_ste_2lvl(uint64_t strtab_base, uint64_t l1ptr, int l1_ste_offset, uint64_t l2ptr, int l2_ste_offset, int max_l2_ste) "strtab_base:0x%"PRIx64" l1ptr:0x%"PRIx64" l1_off:0x%x, l2ptr:0x%"PRIx64" l2_off:0x%x max_l2_ste:%d"
 smmuv3_get_ste(uint64_t addr) "STE addr: 0x%"PRIx64
-- 
2.43.0

[PATCH v7 5/5] hw/arm/smmuv3-accel: Read and propagate host vIOMMU events

[Shameer Kolothum]

Install an event handler on the vEVENTQ fd to read and propagate host
generated vIOMMU events to the guest.

The handler runs in QEMU's main loop, using a non-blocking fd registered
via qemu_set_fd_handler().

Reviewed-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Shameer Kolothum <skolothumtho@nvidia.com>
---
 hw/arm/smmuv3-accel.c | 64 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 64 insertions(+)

diff --git a/hw/arm/smmuv3-accel.c b/hw/arm/smmuv3-accel.c
index f703ea1aac..17306cd04b 100644
--- a/hw/arm/smmuv3-accel.c
+++ b/hw/arm/smmuv3-accel.c
@@ -390,6 +390,50 @@ bool smmuv3_accel_issue_inv_cmd(SMMUv3State *bs, void *cmd, SMMUDevice *sdev,
                    sizeof(Cmd), &entry_num, cmd, errp);
 }
 
+static void smmuv3_accel_event_read(void *opaque)
+{
+    SMMUv3State *s = opaque;
+    IOMMUFDVeventq *veventq = s->s_accel->veventq;
+    struct {
+        struct iommufd_vevent_header hdr;
+        struct iommu_vevent_arm_smmuv3 vevent;
+    } buf;
+    enum iommu_veventq_type type = IOMMU_VEVENTQ_TYPE_ARM_SMMUV3;
+    uint32_t id = veventq->veventq_id;
+    uint32_t last_seq = veventq->last_event_seq;
+    ssize_t bytes;
+
+    bytes = read(veventq->veventq_fd, &buf, sizeof(buf));
+    if (bytes <= 0) {
+        if (errno == EAGAIN || errno == EINTR) {
+            return;
+        }
+        error_report_once("vEVENTQ(type %u id %u): read failed (%m)", type, id);
+        return;
+    }
+
+    if (bytes == sizeof(buf.hdr) &&
+        (buf.hdr.flags & IOMMU_VEVENTQ_FLAG_LOST_EVENTS)) {
+        error_report_once("vEVENTQ(type %u id %u): overflowed", type, id);
+        veventq->event_start = false;
+        return;
+    }
+    if (bytes < sizeof(buf)) {
+        error_report_once("vEVENTQ(type %u id %u): short read(%zd/%zd bytes)",
+                          type, id, bytes, sizeof(buf));
+        return;
+    }
+
+    /* Check sequence in hdr for lost events if any */
+    if (veventq->event_start && (buf.hdr.sequence - last_seq != 1)) {
+        error_report_once("vEVENTQ(type %u id %u): lost %u event(s)",
+                          type, id, buf.hdr.sequence - last_seq - 1);
+    }
+    veventq->last_event_seq = buf.hdr.sequence;
+    veventq->event_start = true;
+    smmuv3_propagate_event(s, (Evt *)&buf.vevent);
+}
+
 static void smmuv3_accel_free_veventq(SMMUv3AccelState *accel)
 {
     IOMMUFDVeventq *veventq = accel->veventq;
@@ -397,6 +441,7 @@ static void smmuv3_accel_free_veventq(SMMUv3AccelState *accel)
     if (!veventq) {
         return;
     }
+    qemu_set_fd_handler(veventq->veventq_fd, NULL, NULL, NULL);
     close(veventq->veventq_fd);
     iommufd_backend_free_id(accel->viommu->iommufd, veventq->veventq_id);
     g_free(veventq);
@@ -424,6 +469,7 @@ bool smmuv3_accel_alloc_veventq(SMMUv3State *s, Error **errp)
     IOMMUFDVeventq *veventq;
     uint32_t veventq_id;
     uint32_t veventq_fd;
+    int flags;
 
     if (!accel || !accel->viommu) {
         return true;
@@ -445,12 +491,30 @@ bool smmuv3_accel_alloc_veventq(SMMUv3State *s, Error **errp)
         return false;
     }
 
+    flags = fcntl(veventq_fd, F_GETFL);
+    if (flags < 0) {
+        error_setg_errno(errp, errno, "Failed to get flags for vEVENTQ fd");
+        goto free_veventq;
+    }
+    if (fcntl(veventq_fd, F_SETFL, flags | O_NONBLOCK) < 0) {
+        error_setg_errno(errp, errno, "Failed to set O_NONBLOCK on vEVENTQ fd");
+        goto free_veventq;
+    }
+
     veventq = g_new0(IOMMUFDVeventq, 1);
     veventq->veventq_id = veventq_id;
     veventq->veventq_fd = veventq_fd;
     veventq->viommu = accel->viommu;
     accel->veventq = veventq;
+
+    /* Set up event handler for veventq fd */
+    qemu_set_fd_handler(veventq_fd, smmuv3_accel_event_read, NULL, s);
     return true;
+
+free_veventq:
+    close(veventq_fd);
+    iommufd_backend_free_id(accel->viommu->iommufd, veventq_id);
+    return false;
 }
 
 static bool
-- 
2.43.0

Re: [PATCH v7 0/5] vEVENTQ support for accelerated SMMUv3 devices

[Eric Auger]

Hi,

On 2/19/26 10:00 AM, Shameer Kolothum wrote:
> Hi,
>
> Changes from v6:
>  https://lore.kernel.org/qemu-devel/20260213103942.142823-1-skolothumtho@nvidia.com/
>  
>  -Addressed feedback and picked up R-by/T-by tags. Thanks!
>  -Updated error report (patch #3)
>
> Please take a look and let me know.
Thanks Shameer, this looks ready now!

Feel free to grab my

Tested-by: Eric Auger <eric.auger@redhat.com>

Eric
>
> Thanks,
> Shameer
>
> ---
> Background(From v2):
>
> When accel=on is enabled for an SMMUv3 instance, the host hardware SMMUv3
> may generate Stage-1 (S1) event notifications that are intended for the
> vIOMMU instance in userspace.
>
> This series adds QEMU support for receiving such host events through the
> vEVENTQ interface and propagating them to the guest. The implementation
> uses the vEVENTQ support provided by the IOMMUFD subsystem in the kernel.
>
> I have tested this on a Grace platform using some hacks to generate
> events. Further testing and feedback are welcome.
> ---
>
>
> Nicolin Chen (2):
>   backends/iommufd: Introduce iommufd_backend_alloc_veventq
>   hw/arm/smmuv3-accel: Allocate vEVENTQ for accelerated SMMUv3 devices
>
> Shameer Kolothum (3):
>   hw/arm/smmuv3-accel: Add viommu free helper
>   hw/arm/smmuv3: Introduce a helper function for event propagation
>   hw/arm/smmuv3-accel: Read and propagate host vIOMMU events
>
>  hw/arm/smmuv3-accel.h    |   6 ++
>  hw/arm/smmuv3-internal.h |   4 ++
>  include/system/iommufd.h |  14 ++++
>  backends/iommufd.c       |  31 +++++++++
>  hw/arm/smmuv3-accel.c    | 146 ++++++++++++++++++++++++++++++++++++---
>  hw/arm/smmuv3.c          |  26 +++++--
>  backends/trace-events    |   1 +
>  hw/arm/trace-events      |   2 +-
>  8 files changed, 215 insertions(+), 15 deletions(-)
>

Re: [PATCH v7 1/5] backends/iommufd: Introduce iommufd_backend_alloc_veventq

[Jonathan Cameron via]

On Thu, 19 Feb 2026 09:00:59 +0000
Shameer Kolothum <skolothumtho@nvidia.com> wrote:

> From: Nicolin Chen <nicolinc@nvidia.com>
> 
> Add a new helper for IOMMU_VEVENTQ_ALLOC ioctl to allocate a virtual event
> queue (vEVENTQ) for a vIOMMU object, and define struct IOMMUFDVeventq to
> represent the allocated vEVENTQ.
> 
> Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
> Reviewed-by: Eric Auger <eric.auger@redhat.com>
> Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Shameer Kolothum <skolothumtho@nvidia.com>
Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>

Re: [PATCH v7 1/5] backends/iommufd: Introduce iommufd_backend_alloc_veventq

[Jonathan Cameron via qemu development]

On Thu, 19 Feb 2026 09:00:59 +0000
Shameer Kolothum <skolothumtho@nvidia.com> wrote:

> From: Nicolin Chen <nicolinc@nvidia.com>
> 
> Add a new helper for IOMMU_VEVENTQ_ALLOC ioctl to allocate a virtual event
> queue (vEVENTQ) for a vIOMMU object, and define struct IOMMUFDVeventq to
> represent the allocated vEVENTQ.
> 
> Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
> Reviewed-by: Eric Auger <eric.auger@redhat.com>
> Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Shameer Kolothum <skolothumtho@nvidia.com>
Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>

Re: [PATCH v7 2/5] hw/arm/smmuv3-accel: Add viommu free helper

[Jonathan Cameron via]

On Thu, 19 Feb 2026 09:01:00 +0000
Shameer Kolothum <skolothumtho@nvidia.com> wrote:

> Move viommu teardown into a helper function and use it from the
> last device removal path. No functional change.

Minor thing, but this description should really include 'why' factoring this
out makes sense. I was kind of assuming reuse, but there doesn't seem to be
any in this series.  So I guess just grouping related code for readability?

> 
> Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
> Reviewed-by: Eric Auger <eric.auger@redhat.com>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Shameer Kolothum <skolothumtho@nvidia.com>
Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>

Re: [PATCH v7 2/5] hw/arm/smmuv3-accel: Add viommu free helper

[Jonathan Cameron via qemu development]

On Thu, 19 Feb 2026 09:01:00 +0000
Shameer Kolothum <skolothumtho@nvidia.com> wrote:

> Move viommu teardown into a helper function and use it from the
> last device removal path. No functional change.

Minor thing, but this description should really include 'why' factoring this
out makes sense. I was kind of assuming reuse, but there doesn't seem to be
any in this series.  So I guess just grouping related code for readability?

> 
> Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
> Reviewed-by: Eric Auger <eric.auger@redhat.com>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Shameer Kolothum <skolothumtho@nvidia.com>
Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>

Re: [PATCH v7 3/5] hw/arm/smmuv3-accel: Allocate vEVENTQ for accelerated SMMUv3 devices

[Jonathan Cameron via]

On Thu, 19 Feb 2026 09:01:01 +0000
Shameer Kolothum <skolothumtho@nvidia.com> wrote:

> From: Nicolin Chen <nicolinc@nvidia.com>
> 
> When the guest enables the Event Queue and a vIOMMU is present, allocate a
> vEVENTQ object so that host-side events related to the vIOMMU can be
> received and propagated back to the guest.
> 
> For cold-plugged devices using SMMUv3 acceleration, the vIOMMU is created
> before the guest boots. In this case, the vEVENTQ is allocated when the
> guest writes to SMMU_CR0 and sets EVENTQEN = 1.
> 
> If no cold-plugged device exists at boot (i.e. no vIOMMU initially), the
> vEVENTQ is allocated when a vIOMMU is created, i.e. during the first
> device hot-plug.

Why do it differently?

> 
> Errors from command queue consumption and vEVENTQ allocation are reported
> independently as the two operations are unrelated.
> 
> Event read and propagation will be added in a later patch.
> 
> Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
> Reviewed-by: Eric Auger <eric.auger@redhat.com>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Shameer Kolothum <skolothumtho@nvidia.com>

Re: [PATCH v7 3/5] hw/arm/smmuv3-accel: Allocate vEVENTQ for accelerated SMMUv3 devices

[Jonathan Cameron via qemu development]

On Thu, 19 Feb 2026 09:01:01 +0000
Shameer Kolothum <skolothumtho@nvidia.com> wrote:

> From: Nicolin Chen <nicolinc@nvidia.com>
> 
> When the guest enables the Event Queue and a vIOMMU is present, allocate a
> vEVENTQ object so that host-side events related to the vIOMMU can be
> received and propagated back to the guest.
> 
> For cold-plugged devices using SMMUv3 acceleration, the vIOMMU is created
> before the guest boots. In this case, the vEVENTQ is allocated when the
> guest writes to SMMU_CR0 and sets EVENTQEN = 1.
> 
> If no cold-plugged device exists at boot (i.e. no vIOMMU initially), the
> vEVENTQ is allocated when a vIOMMU is created, i.e. during the first
> device hot-plug.

Why do it differently?

> 
> Errors from command queue consumption and vEVENTQ allocation are reported
> independently as the two operations are unrelated.
> 
> Event read and propagation will be added in a later patch.
> 
> Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
> Reviewed-by: Eric Auger <eric.auger@redhat.com>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Shameer Kolothum <skolothumtho@nvidia.com>

Re: [PATCH v7 4/5] hw/arm/smmuv3: Introduce a helper function for event propagation

[Jonathan Cameron via]

On Thu, 19 Feb 2026 09:01:02 +0000
Shameer Kolothum <skolothumtho@nvidia.com> wrote:

> Factor out the code that propagates event records to the guest into a
> helper function. The accelerated SMMUv3 path can use this to propagate
> host events in a subsequent patch.
> 
> Take the mutex inside the helper before accessing the Event Queue.
> Today event propagation occurs only in the core SMMUv3 path and is
> effectively serialized. A subsequent patch will also invoke this helper
> from the accelerated event read path, which may run concurrently.
> Therefore serialization is required here.
> 
> No functional change intended.
> 
> Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
> Reviewed-by: Eric Auger <eric.auger@redhat.com>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Shameer Kolothum <skolothumtho@nvidia.com>
Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>

Re: [PATCH v7 4/5] hw/arm/smmuv3: Introduce a helper function for event propagation

[Jonathan Cameron via qemu development]

On Thu, 19 Feb 2026 09:01:02 +0000
Shameer Kolothum <skolothumtho@nvidia.com> wrote:

> Factor out the code that propagates event records to the guest into a
> helper function. The accelerated SMMUv3 path can use this to propagate
> host events in a subsequent patch.
> 
> Take the mutex inside the helper before accessing the Event Queue.
> Today event propagation occurs only in the core SMMUv3 path and is
> effectively serialized. A subsequent patch will also invoke this helper
> from the accelerated event read path, which may run concurrently.
> Therefore serialization is required here.
> 
> No functional change intended.
> 
> Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
> Reviewed-by: Eric Auger <eric.auger@redhat.com>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Shameer Kolothum <skolothumtho@nvidia.com>
Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>

Re: [PATCH v7 5/5] hw/arm/smmuv3-accel: Read and propagate host vIOMMU events

[Jonathan Cameron via]

On Thu, 19 Feb 2026 09:01:03 +0000
Shameer Kolothum <skolothumtho@nvidia.com> wrote:

> Install an event handler on the vEVENTQ fd to read and propagate host
> generated vIOMMU events to the guest.
> 
> The handler runs in QEMU's main loop, using a non-blocking fd registered
> via qemu_set_fd_handler().
> 
> Reviewed-by: Eric Auger <eric.auger@redhat.com>
> Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Shameer Kolothum <skolothumtho@nvidia.com>
Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>

Re: [PATCH v7 5/5] hw/arm/smmuv3-accel: Read and propagate host vIOMMU events

[Jonathan Cameron via qemu development]

On Thu, 19 Feb 2026 09:01:03 +0000
Shameer Kolothum <skolothumtho@nvidia.com> wrote:

> Install an event handler on the vEVENTQ fd to read and propagate host
> generated vIOMMU events to the guest.
> 
> The handler runs in QEMU's main loop, using a non-blocking fd registered
> via qemu_set_fd_handler().
> 
> Reviewed-by: Eric Auger <eric.auger@redhat.com>
> Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Shameer Kolothum <skolothumtho@nvidia.com>
Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>

RE: [PATCH v7 3/5] hw/arm/smmuv3-accel: Allocate vEVENTQ for accelerated SMMUv3 devices

[Shameer Kolothum Thodi]

> -----Original Message-----
> From: Jonathan Cameron <jonathan.cameron@huawei.com>
> Sent: 24 February 2026 14:29
> To: Shameer Kolothum Thodi <skolothumtho@nvidia.com>
> Cc: qemu-arm@nongnu.org; qemu-devel@nongnu.org;
> eric.auger@redhat.com; peter.maydell@linaro.org; Nicolin Chen
> <nicolinc@nvidia.com>; Nathan Chen <nathanc@nvidia.com>; Matt Ochs
> <mochs@nvidia.com>; Jiandi An <jan@nvidia.com>; Jason Gunthorpe
> <jgg@nvidia.com>; zhangfei.gao@linaro.org; zhenzhong.duan@intel.com;
> Krishnakant Jaju <kjaju@nvidia.com>
> Subject: Re: [PATCH v7 3/5] hw/arm/smmuv3-accel: Allocate vEVENTQ for
> accelerated SMMUv3 devices
> 
> External email: Use caution opening links or attachments
> 
> 
> On Thu, 19 Feb 2026 09:01:01 +0000
> Shameer Kolothum <skolothumtho@nvidia.com> wrote:
> 
> > From: Nicolin Chen <nicolinc@nvidia.com>
> >
> > When the guest enables the Event Queue and a vIOMMU is present,
> > allocate a vEVENTQ object so that host-side events related to the
> > vIOMMU can be received and propagated back to the guest.
> >
> > For cold-plugged devices using SMMUv3 acceleration, the vIOMMU is
> > created before the guest boots. In this case, the vEVENTQ is allocated
> > when the guest writes to SMMU_CR0 and sets EVENTQEN = 1.
> >
> > If no cold-plugged device exists at boot (i.e. no vIOMMU initially),
> > the vEVENTQ is allocated when a vIOMMU is created, i.e. during the
> > first device hot-plug.
> 
> Why do it differently?

This is because a vEVENTQ must be associated with a vIOMMU, and
iommufd_backend_alloc_veventq() requires a valid viommu_id.

A vIOMMU is only created once at least one vfio-pci device is
attached. If the VM boots without any cold-plugged devices, no
vIOMMU exists at that point.

In that case, the vIOMMU is created during the first device
hot-plug, so we allocate the vEVENTQ at that time instead.

So, the difference in behaviour simply follows when the vIOMMU
becomes available.

Hope that is clear now.

Thanks,
Shameer

RE: [PATCH v7 2/5] hw/arm/smmuv3-accel: Add viommu free helper

[Shameer Kolothum Thodi]

> -----Original Message-----
> From: Jonathan Cameron <jonathan.cameron@huawei.com>
> Sent: 24 February 2026 14:14
> To: Shameer Kolothum Thodi <skolothumtho@nvidia.com>
> Cc: qemu-arm@nongnu.org; qemu-devel@nongnu.org;
> eric.auger@redhat.com; peter.maydell@linaro.org; Nicolin Chen
> <nicolinc@nvidia.com>; Nathan Chen <nathanc@nvidia.com>; Matt Ochs
> <mochs@nvidia.com>; Jiandi An <jan@nvidia.com>; Jason Gunthorpe
> <jgg@nvidia.com>; zhangfei.gao@linaro.org; zhenzhong.duan@intel.com;
> Krishnakant Jaju <kjaju@nvidia.com>
> Subject: Re: [PATCH v7 2/5] hw/arm/smmuv3-accel: Add viommu free helper
> 
> External email: Use caution opening links or attachments
> 
> 
> On Thu, 19 Feb 2026 09:01:00 +0000
> Shameer Kolothum <skolothumtho@nvidia.com> wrote:
> 
> > Move viommu teardown into a helper function and use it from the
> > last device removal path. No functional change.
> 
> Minor thing, but this description should really include 'why' factoring this
> out makes sense. I was kind of assuming reuse, but there doesn't seem to be
> any in this series.  So I guess just grouping related code for readability?

Right, this is primarily for grouping related teardown logic and
improving readability.

It also provides a single place to extend the teardown in future,
e.g., when freeing related objects such as vEVENTQ.

> 
> >
> > Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
> > Reviewed-by: Eric Auger <eric.auger@redhat.com>
> > Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> > Signed-off-by: Shameer Kolothum <skolothumtho@nvidia.com>
> Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>

Thanks,
Shameer

Re: [PATCH v7 3/5] hw/arm/smmuv3-accel: Allocate vEVENTQ for accelerated SMMUv3 devices

[Jonathan Cameron via]

On Tue, 24 Feb 2026 14:41:08 +0000
Shameer Kolothum Thodi <skolothumtho@nvidia.com> wrote:

> > -----Original Message-----
> > From: Jonathan Cameron <jonathan.cameron@huawei.com>
> > Sent: 24 February 2026 14:29
> > To: Shameer Kolothum Thodi <skolothumtho@nvidia.com>
> > Cc: qemu-arm@nongnu.org; qemu-devel@nongnu.org;
> > eric.auger@redhat.com; peter.maydell@linaro.org; Nicolin Chen
> > <nicolinc@nvidia.com>; Nathan Chen <nathanc@nvidia.com>; Matt Ochs
> > <mochs@nvidia.com>; Jiandi An <jan@nvidia.com>; Jason Gunthorpe
> > <jgg@nvidia.com>; zhangfei.gao@linaro.org; zhenzhong.duan@intel.com;
> > Krishnakant Jaju <kjaju@nvidia.com>
> > Subject: Re: [PATCH v7 3/5] hw/arm/smmuv3-accel: Allocate vEVENTQ for
> > accelerated SMMUv3 devices
> > 
> > External email: Use caution opening links or attachments
> > 
> > 
> > On Thu, 19 Feb 2026 09:01:01 +0000
> > Shameer Kolothum <skolothumtho@nvidia.com> wrote:
> >   
> > > From: Nicolin Chen <nicolinc@nvidia.com>
> > >
> > > When the guest enables the Event Queue and a vIOMMU is present,
> > > allocate a vEVENTQ object so that host-side events related to the
> > > vIOMMU can be received and propagated back to the guest.
> > >
> > > For cold-plugged devices using SMMUv3 acceleration, the vIOMMU is
> > > created before the guest boots. In this case, the vEVENTQ is allocated
> > > when the guest writes to SMMU_CR0 and sets EVENTQEN = 1.
> > >
> > > If no cold-plugged device exists at boot (i.e. no vIOMMU initially),
> > > the vEVENTQ is allocated when a vIOMMU is created, i.e. during the
> > > first device hot-plug.  
> > 
> > Why do it differently?  
> 
> This is because a vEVENTQ must be associated with a vIOMMU, and
> iommufd_backend_alloc_veventq() requires a valid viommu_id.
> 
> A vIOMMU is only created once at least one vfio-pci device is
> attached. If the VM boots without any cold-plugged devices, no
> vIOMMU exists at that point.
> 
> In that case, the vIOMMU is created during the first device
> hot-plug, so we allocate the vEVENTQ at that time instead.
> 
> So, the difference in behaviour simply follows when the vIOMMU
> becomes available.
> 
> Hope that is clear now.
Ok.  So maybe this is just a description thing that was confusing me.
The key to my mental model is that both conditions must be met. That
is SMMU_CR0 has EVENTQEN = 1 set and there is a VFIO device present.
They happen in different orders in the two cases?

What guarantees we don't get a hotplug event before the guest has
set EVENTQEN = 1?

Jonathan

> 
> Thanks,
> Shameer

Re: [PATCH v7 3/5] hw/arm/smmuv3-accel: Allocate vEVENTQ for accelerated SMMUv3 devices

[Jonathan Cameron via qemu development]

On Tue, 24 Feb 2026 14:41:08 +0000
Shameer Kolothum Thodi <skolothumtho@nvidia.com> wrote:

> > -----Original Message-----
> > From: Jonathan Cameron <jonathan.cameron@huawei.com>
> > Sent: 24 February 2026 14:29
> > To: Shameer Kolothum Thodi <skolothumtho@nvidia.com>
> > Cc: qemu-arm@nongnu.org; qemu-devel@nongnu.org;
> > eric.auger@redhat.com; peter.maydell@linaro.org; Nicolin Chen
> > <nicolinc@nvidia.com>; Nathan Chen <nathanc@nvidia.com>; Matt Ochs
> > <mochs@nvidia.com>; Jiandi An <jan@nvidia.com>; Jason Gunthorpe
> > <jgg@nvidia.com>; zhangfei.gao@linaro.org; zhenzhong.duan@intel.com;
> > Krishnakant Jaju <kjaju@nvidia.com>
> > Subject: Re: [PATCH v7 3/5] hw/arm/smmuv3-accel: Allocate vEVENTQ for
> > accelerated SMMUv3 devices
> > 
> > External email: Use caution opening links or attachments
> > 
> > 
> > On Thu, 19 Feb 2026 09:01:01 +0000
> > Shameer Kolothum <skolothumtho@nvidia.com> wrote:
> >   
> > > From: Nicolin Chen <nicolinc@nvidia.com>
> > >
> > > When the guest enables the Event Queue and a vIOMMU is present,
> > > allocate a vEVENTQ object so that host-side events related to the
> > > vIOMMU can be received and propagated back to the guest.
> > >
> > > For cold-plugged devices using SMMUv3 acceleration, the vIOMMU is
> > > created before the guest boots. In this case, the vEVENTQ is allocated
> > > when the guest writes to SMMU_CR0 and sets EVENTQEN = 1.
> > >
> > > If no cold-plugged device exists at boot (i.e. no vIOMMU initially),
> > > the vEVENTQ is allocated when a vIOMMU is created, i.e. during the
> > > first device hot-plug.  
> > 
> > Why do it differently?  
> 
> This is because a vEVENTQ must be associated with a vIOMMU, and
> iommufd_backend_alloc_veventq() requires a valid viommu_id.
> 
> A vIOMMU is only created once at least one vfio-pci device is
> attached. If the VM boots without any cold-plugged devices, no
> vIOMMU exists at that point.
> 
> In that case, the vIOMMU is created during the first device
> hot-plug, so we allocate the vEVENTQ at that time instead.
> 
> So, the difference in behaviour simply follows when the vIOMMU
> becomes available.
> 
> Hope that is clear now.
Ok.  So maybe this is just a description thing that was confusing me.
The key to my mental model is that both conditions must be met. That
is SMMU_CR0 has EVENTQEN = 1 set and there is a VFIO device present.
They happen in different orders in the two cases?

What guarantees we don't get a hotplug event before the guest has
set EVENTQEN = 1?

Jonathan

> 
> Thanks,
> Shameer

RE: [PATCH v7 3/5] hw/arm/smmuv3-accel: Allocate vEVENTQ for accelerated SMMUv3 devices

[Shameer Kolothum Thodi]

> -----Original Message-----
> From: Jonathan Cameron <jonathan.cameron@huawei.com>
> Sent: 24 February 2026 15:02
> To: Shameer Kolothum Thodi <skolothumtho@nvidia.com>
> Cc: qemu-arm@nongnu.org; qemu-devel@nongnu.org;
> eric.auger@redhat.com; peter.maydell@linaro.org; Nicolin Chen
> <nicolinc@nvidia.com>; Nathan Chen <nathanc@nvidia.com>; Matt Ochs
> <mochs@nvidia.com>; Jiandi An <jan@nvidia.com>; Jason Gunthorpe
> <jgg@nvidia.com>; zhangfei.gao@linaro.org; zhenzhong.duan@intel.com;
> Krishnakant Jaju <kjaju@nvidia.com>
> Subject: Re: [PATCH v7 3/5] hw/arm/smmuv3-accel: Allocate vEVENTQ for
> accelerated SMMUv3 devices
> 
> External email: Use caution opening links or attachments
> 
> 
> On Tue, 24 Feb 2026 14:41:08 +0000
> Shameer Kolothum Thodi <skolothumtho@nvidia.com> wrote:
> 
> > > -----Original Message-----
> > > From: Jonathan Cameron <jonathan.cameron@huawei.com>
> > > Sent: 24 February 2026 14:29
> > > To: Shameer Kolothum Thodi <skolothumtho@nvidia.com>
> > > Cc: qemu-arm@nongnu.org; qemu-devel@nongnu.org;
> > > eric.auger@redhat.com; peter.maydell@linaro.org; Nicolin Chen
> > > <nicolinc@nvidia.com>; Nathan Chen <nathanc@nvidia.com>; Matt Ochs
> > > <mochs@nvidia.com>; Jiandi An <jan@nvidia.com>; Jason Gunthorpe
> > > <jgg@nvidia.com>; zhangfei.gao@linaro.org; zhenzhong.duan@intel.com;
> > > Krishnakant Jaju <kjaju@nvidia.com>
> > > Subject: Re: [PATCH v7 3/5] hw/arm/smmuv3-accel: Allocate vEVENTQ for
> > > accelerated SMMUv3 devices
> > >
> > > External email: Use caution opening links or attachments
> > >
> > >
> > > On Thu, 19 Feb 2026 09:01:01 +0000
> > > Shameer Kolothum <skolothumtho@nvidia.com> wrote:
> > >
> > > > From: Nicolin Chen <nicolinc@nvidia.com>
> > > >
> > > > When the guest enables the Event Queue and a vIOMMU is present,
> > > > allocate a vEVENTQ object so that host-side events related to the
> > > > vIOMMU can be received and propagated back to the guest.
> > > >
> > > > For cold-plugged devices using SMMUv3 acceleration, the vIOMMU is
> > > > created before the guest boots. In this case, the vEVENTQ is allocated
> > > > when the guest writes to SMMU_CR0 and sets EVENTQEN = 1.
> > > >
> > > > If no cold-plugged device exists at boot (i.e. no vIOMMU initially),
> > > > the vEVENTQ is allocated when a vIOMMU is created, i.e. during the
> > > > first device hot-plug.
> > >
> > > Why do it differently?
> >
> > This is because a vEVENTQ must be associated with a vIOMMU, and
> > iommufd_backend_alloc_veventq() requires a valid viommu_id.
> >
> > A vIOMMU is only created once at least one vfio-pci device is
> > attached. If the VM boots without any cold-plugged devices, no
> > vIOMMU exists at that point.
> >
> > In that case, the vIOMMU is created during the first device
> > hot-plug, so we allocate the vEVENTQ at that time instead.
> >
> > So, the difference in behaviour simply follows when the vIOMMU
> > becomes available.
> >
> > Hope that is clear now.
> Ok.  So maybe this is just a description thing that was confusing me.
> The key to my mental model is that both conditions must be met. That
> is SMMU_CR0 has EVENTQEN = 1 set and there is a VFIO device present.
> They happen in different orders in the two cases?

Yes.  And EVENTQEN is set by Guest ARM SMMUv3 driver probe time.

> 
> What guarantees we don't get a hotplug event before the guest has
> set EVENTQEN = 1?

 vEVENTQ needs two things:

1) The guest must set EVENTQEN=1.
2) A vIOMMU must already exist (which happens when a VFIO device is attached).

EVENTQEN=1 is set by the guest when the SMMUv3 driver probes.

These two things can happen in either order.

In the cold-plug case, the vIOMMU already exists before the guest driver
probes. So when the guest sets EVENTQEN=1 during driver probe, we
allocate the vEVENTQ at that time.

In the hot-plug-first case, the first VFIO hotplug creates the vIOMMU,
possibly before the guest has loaded the SMMUv3 driver. In that case,
we wait until the guest sets EVENTQEN=1 during driver probe, and then
allocate the vEVENTQ.

So we do not depend on ordering. We just check that both conditions
are true, and allocate when the second one becomes true.

Thanks,
Shameer

Re: [PATCH v7 3/5] hw/arm/smmuv3-accel: Allocate vEVENTQ for accelerated SMMUv3 devices

[Jonathan Cameron via]

On Tue, 24 Feb 2026 15:18:00 +0000
Shameer Kolothum Thodi <skolothumtho@nvidia.com> wrote:

> > -----Original Message-----
> > From: Jonathan Cameron <jonathan.cameron@huawei.com>
> > Sent: 24 February 2026 15:02
> > To: Shameer Kolothum Thodi <skolothumtho@nvidia.com>
> > Cc: qemu-arm@nongnu.org; qemu-devel@nongnu.org;
> > eric.auger@redhat.com; peter.maydell@linaro.org; Nicolin Chen
> > <nicolinc@nvidia.com>; Nathan Chen <nathanc@nvidia.com>; Matt Ochs
> > <mochs@nvidia.com>; Jiandi An <jan@nvidia.com>; Jason Gunthorpe
> > <jgg@nvidia.com>; zhangfei.gao@linaro.org; zhenzhong.duan@intel.com;
> > Krishnakant Jaju <kjaju@nvidia.com>
> > Subject: Re: [PATCH v7 3/5] hw/arm/smmuv3-accel: Allocate vEVENTQ for
> > accelerated SMMUv3 devices
> > 
> > External email: Use caution opening links or attachments
> > 
> > 
> > On Tue, 24 Feb 2026 14:41:08 +0000
> > Shameer Kolothum Thodi <skolothumtho@nvidia.com> wrote:
> >   
> > > > -----Original Message-----
> > > > From: Jonathan Cameron <jonathan.cameron@huawei.com>
> > > > Sent: 24 February 2026 14:29
> > > > To: Shameer Kolothum Thodi <skolothumtho@nvidia.com>
> > > > Cc: qemu-arm@nongnu.org; qemu-devel@nongnu.org;
> > > > eric.auger@redhat.com; peter.maydell@linaro.org; Nicolin Chen
> > > > <nicolinc@nvidia.com>; Nathan Chen <nathanc@nvidia.com>; Matt Ochs
> > > > <mochs@nvidia.com>; Jiandi An <jan@nvidia.com>; Jason Gunthorpe
> > > > <jgg@nvidia.com>; zhangfei.gao@linaro.org; zhenzhong.duan@intel.com;
> > > > Krishnakant Jaju <kjaju@nvidia.com>
> > > > Subject: Re: [PATCH v7 3/5] hw/arm/smmuv3-accel: Allocate vEVENTQ for
> > > > accelerated SMMUv3 devices
> > > >
> > > > External email: Use caution opening links or attachments
> > > >
> > > >
> > > > On Thu, 19 Feb 2026 09:01:01 +0000
> > > > Shameer Kolothum <skolothumtho@nvidia.com> wrote:
> > > >  
> > > > > From: Nicolin Chen <nicolinc@nvidia.com>
> > > > >
> > > > > When the guest enables the Event Queue and a vIOMMU is present,
> > > > > allocate a vEVENTQ object so that host-side events related to the
> > > > > vIOMMU can be received and propagated back to the guest.
> > > > >
> > > > > For cold-plugged devices using SMMUv3 acceleration, the vIOMMU is
> > > > > created before the guest boots. In this case, the vEVENTQ is allocated
> > > > > when the guest writes to SMMU_CR0 and sets EVENTQEN = 1.
> > > > >
> > > > > If no cold-plugged device exists at boot (i.e. no vIOMMU initially),
> > > > > the vEVENTQ is allocated when a vIOMMU is created, i.e. during the
> > > > > first device hot-plug.  
> > > >
> > > > Why do it differently?  
> > >
> > > This is because a vEVENTQ must be associated with a vIOMMU, and
> > > iommufd_backend_alloc_veventq() requires a valid viommu_id.
> > >
> > > A vIOMMU is only created once at least one vfio-pci device is
> > > attached. If the VM boots without any cold-plugged devices, no
> > > vIOMMU exists at that point.
> > >
> > > In that case, the vIOMMU is created during the first device
> > > hot-plug, so we allocate the vEVENTQ at that time instead.
> > >
> > > So, the difference in behaviour simply follows when the vIOMMU
> > > becomes available.
> > >
> > > Hope that is clear now.  
> > Ok.  So maybe this is just a description thing that was confusing me.
> > The key to my mental model is that both conditions must be met. That
> > is SMMU_CR0 has EVENTQEN = 1 set and there is a VFIO device present.
> > They happen in different orders in the two cases?  
> 
> Yes.  And EVENTQEN is set by Guest ARM SMMUv3 driver probe time.
> 
> > 
> > What guarantees we don't get a hotplug event before the guest has
> > set EVENTQEN = 1?  
> 
>  vEVENTQ needs two things:
> 
> 1) The guest must set EVENTQEN=1.
> 2) A vIOMMU must already exist (which happens when a VFIO device is attached).
> 
> EVENTQEN=1 is set by the guest when the SMMUv3 driver probes.
> 
> These two things can happen in either order.
> 
> In the cold-plug case, the vIOMMU already exists before the guest driver
> probes. So when the guest sets EVENTQEN=1 during driver probe, we
> allocate the vEVENTQ at that time.
> 
> In the hot-plug-first case, the first VFIO hotplug creates the vIOMMU,
> possibly before the guest has loaded the SMMUv3 driver. In that case,
> we wait until the guest sets EVENTQEN=1 during driver probe, and then
> allocate the vEVENTQ.
> 
> So we do not depend on ordering. We just check that both conditions
> are true, and allocate when the second one becomes true.

Perfect.  That explanation works for me. If you respin maybe start
the explanation with the full conditions that must be met, then break
down why they occur in different orders in the cold and hotplug
(after we reach a particular point in guest boot) cases.

Jonathan

> 
> Thanks,
> Shameer
> 

Re: [PATCH v7 3/5] hw/arm/smmuv3-accel: Allocate vEVENTQ for accelerated SMMUv3 devices

[Jonathan Cameron via qemu development]

On Tue, 24 Feb 2026 15:18:00 +0000
Shameer Kolothum Thodi <skolothumtho@nvidia.com> wrote:

> > -----Original Message-----
> > From: Jonathan Cameron <jonathan.cameron@huawei.com>
> > Sent: 24 February 2026 15:02
> > To: Shameer Kolothum Thodi <skolothumtho@nvidia.com>
> > Cc: qemu-arm@nongnu.org; qemu-devel@nongnu.org;
> > eric.auger@redhat.com; peter.maydell@linaro.org; Nicolin Chen
> > <nicolinc@nvidia.com>; Nathan Chen <nathanc@nvidia.com>; Matt Ochs
> > <mochs@nvidia.com>; Jiandi An <jan@nvidia.com>; Jason Gunthorpe
> > <jgg@nvidia.com>; zhangfei.gao@linaro.org; zhenzhong.duan@intel.com;
> > Krishnakant Jaju <kjaju@nvidia.com>
> > Subject: Re: [PATCH v7 3/5] hw/arm/smmuv3-accel: Allocate vEVENTQ for
> > accelerated SMMUv3 devices
> > 
> > External email: Use caution opening links or attachments
> > 
> > 
> > On Tue, 24 Feb 2026 14:41:08 +0000
> > Shameer Kolothum Thodi <skolothumtho@nvidia.com> wrote:
> >   
> > > > -----Original Message-----
> > > > From: Jonathan Cameron <jonathan.cameron@huawei.com>
> > > > Sent: 24 February 2026 14:29
> > > > To: Shameer Kolothum Thodi <skolothumtho@nvidia.com>
> > > > Cc: qemu-arm@nongnu.org; qemu-devel@nongnu.org;
> > > > eric.auger@redhat.com; peter.maydell@linaro.org; Nicolin Chen
> > > > <nicolinc@nvidia.com>; Nathan Chen <nathanc@nvidia.com>; Matt Ochs
> > > > <mochs@nvidia.com>; Jiandi An <jan@nvidia.com>; Jason Gunthorpe
> > > > <jgg@nvidia.com>; zhangfei.gao@linaro.org; zhenzhong.duan@intel.com;
> > > > Krishnakant Jaju <kjaju@nvidia.com>
> > > > Subject: Re: [PATCH v7 3/5] hw/arm/smmuv3-accel: Allocate vEVENTQ for
> > > > accelerated SMMUv3 devices
> > > >
> > > > External email: Use caution opening links or attachments
> > > >
> > > >
> > > > On Thu, 19 Feb 2026 09:01:01 +0000
> > > > Shameer Kolothum <skolothumtho@nvidia.com> wrote:
> > > >  
> > > > > From: Nicolin Chen <nicolinc@nvidia.com>
> > > > >
> > > > > When the guest enables the Event Queue and a vIOMMU is present,
> > > > > allocate a vEVENTQ object so that host-side events related to the
> > > > > vIOMMU can be received and propagated back to the guest.
> > > > >
> > > > > For cold-plugged devices using SMMUv3 acceleration, the vIOMMU is
> > > > > created before the guest boots. In this case, the vEVENTQ is allocated
> > > > > when the guest writes to SMMU_CR0 and sets EVENTQEN = 1.
> > > > >
> > > > > If no cold-plugged device exists at boot (i.e. no vIOMMU initially),
> > > > > the vEVENTQ is allocated when a vIOMMU is created, i.e. during the
> > > > > first device hot-plug.  
> > > >
> > > > Why do it differently?  
> > >
> > > This is because a vEVENTQ must be associated with a vIOMMU, and
> > > iommufd_backend_alloc_veventq() requires a valid viommu_id.
> > >
> > > A vIOMMU is only created once at least one vfio-pci device is
> > > attached. If the VM boots without any cold-plugged devices, no
> > > vIOMMU exists at that point.
> > >
> > > In that case, the vIOMMU is created during the first device
> > > hot-plug, so we allocate the vEVENTQ at that time instead.
> > >
> > > So, the difference in behaviour simply follows when the vIOMMU
> > > becomes available.
> > >
> > > Hope that is clear now.  
> > Ok.  So maybe this is just a description thing that was confusing me.
> > The key to my mental model is that both conditions must be met. That
> > is SMMU_CR0 has EVENTQEN = 1 set and there is a VFIO device present.
> > They happen in different orders in the two cases?  
> 
> Yes.  And EVENTQEN is set by Guest ARM SMMUv3 driver probe time.
> 
> > 
> > What guarantees we don't get a hotplug event before the guest has
> > set EVENTQEN = 1?  
> 
>  vEVENTQ needs two things:
> 
> 1) The guest must set EVENTQEN=1.
> 2) A vIOMMU must already exist (which happens when a VFIO device is attached).
> 
> EVENTQEN=1 is set by the guest when the SMMUv3 driver probes.
> 
> These two things can happen in either order.
> 
> In the cold-plug case, the vIOMMU already exists before the guest driver
> probes. So when the guest sets EVENTQEN=1 during driver probe, we
> allocate the vEVENTQ at that time.
> 
> In the hot-plug-first case, the first VFIO hotplug creates the vIOMMU,
> possibly before the guest has loaded the SMMUv3 driver. In that case,
> we wait until the guest sets EVENTQEN=1 during driver probe, and then
> allocate the vEVENTQ.
> 
> So we do not depend on ordering. We just check that both conditions
> are true, and allocate when the second one becomes true.

Perfect.  That explanation works for me. If you respin maybe start
the explanation with the full conditions that must be met, then break
down why they occur in different orders in the cold and hotplug
(after we reach a particular point in guest boot) cases.

Jonathan

> 
> Thanks,
> Shameer
>