[RFC PATCH v13 0/4] Report vfio-ap configuration changes

[RFC PATCH v13 0/4] Report vfio-ap configuration changes

[Rorie Reyes]

Changelog:
v13:
- added lock to 'vfio_ap_cfg_chg_notifier_handler' in patch 2
- added RBs for patch 2 and 3 from Tony

v12:
- adding locks to 'ap_chsc_sei_nt0_have_event' and 'ap_chsc_sei_nt0_get_event'

v11:
- reverted return type to int for 'ap_chsc_sei_nt0_get_event'
   - files reflected are 'ap.c', 'ap-bridge'h, and 'ap-stub.c'
- using defined variables to represent return 0 and 1 to reflect logical sense
- update documentation for 'ap_chsc_sei_nt0_get_event' to reflect return types

v10:
- Added reviewed by for patch 4 by Tony and Cedric
   - review needed for patch 2 and 3
- changed 'cfg_chg_events' to a static variable
- locked 'cfg_chg_events' using QemuMutex
- removed ';' at the end of the definition for NT0
- returning bools for 'ap_chsc_sei_nt0_get_event' and
'ap_chsc_sei_nt0_have_event'
- updated the header file that contains 'ap_chsc_sei_nt0_get_event' and
'ap_chsc_sei_nt0_have_event' to a bool function
- added documentation explaining the returning bool functions
- whitespace clean up

v9:
- added SPDX licensing to newly created file 'hw/s390x/ap-stub.c'

v8:
- fixed windows cross-compile build
- moved /hw/vfio/ap-stub.c to /hw/s390x/ap-stub.c
- updated the use of stub file to MAINTAINERS to reflect new location
- removed if_false for 'CONFIG_VFIO_AP' statement from /hw/vfio/meson.build
- added if_false for 'CONFIG_VFIO_AP' to use ap-stub.c in /hw/s390x/meson.build
- all those changes still address '--without-default-devices' issue from v5

v7:
- Dropped initial commit for linux-header file vfio.h since I created two new commits
to address the changes made in v6
- Moved patches 6 and 7 to the beginning of the series after dropping the first patch
   - Because I dropped the initial commit for linux-header file vfio.h, I had to add
VFIO_AP_CFG_CHG_IRQ_INDEX
- Resyncing latest to v6.15-rc3
- Still need Thomas Huth's review of v5 changes for patch 6/6

v6:
- Updating the update-linux-headers script to address kernel commit change 8a14
- Update headers to retrieve uapi information for vfio-ap for update to Linux v6.15-rc1
- Still need Thomas Huth's review of v5 changes for patch 7/7 (see below)

v5:
- configuring using the '--without-default-devices' fails when building the code
- created a stub file for functions ap_chsc_sei_nt0_get_event and ap_chsc_sei_nt0_have_event
- add if_false for 'CONFIG_VFIO_AP' use ap-stub.c in meson.build
- add the use of the stub file to MAINTAINERS since it's a new file

v4:
- allocating cfg_chg_event before inserting into the queue
- calling nt0_have_event in if loop to check if there are any
elemenets in the queue, then calling QTAILQ_FIRST when the check
passes
- moving memset() after the check

v3:
- changes that were made to patch 3/5 should have been made in
patch 2/5

v2:
- removed warnings that weren't needed
- added unregister function
- removed whitelines
- changed variable names for consistency
- removed rc variable and returning 1 or 0 outright
- reversed logics for if statements
- using g_free() instead of free()
- replaced hardcoded numeric values by defining them with #define
in the header

--------------------------------------------------------------------------

Rorie Reyes (4):
  hw/vfio/ap: notification handler for AP config changed event
  hw/vfio/ap: store object indicating AP config changed in a queue
  hw/vfio/ap: Storing event information for an AP configuration change
    event
  s390: implementing CHSC SEI for AP config change

 MAINTAINERS                  |  1 +
 hw/s390x/ap-stub.c           | 21 +++++++++
 hw/s390x/meson.build         |  1 +
 hw/vfio/ap.c                 | 88 ++++++++++++++++++++++++++++++++++++
 include/hw/s390x/ap-bridge.h | 39 ++++++++++++++++
 target/s390x/ioinst.c        | 11 ++++-
 6 files changed, 159 insertions(+), 2 deletions(-)
 create mode 100644 hw/s390x/ap-stub.c

-- 
2.48.1

[RFC PATCH v13 1/4] hw/vfio/ap: notification handler for AP config changed event

[Rorie Reyes]

Register an event notifier handler to process AP configuration
change events by queuing the event and generating a CRW to let
the guest know its AP configuration has changed

Signed-off-by: Rorie Reyes <rreyes@linux.ibm.com>
Reviewed-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 hw/vfio/ap.c | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)

diff --git a/hw/vfio/ap.c b/hw/vfio/ap.c
index 785c0a0197..93c74ebedb 100644
--- a/hw/vfio/ap.c
+++ b/hw/vfio/ap.c
@@ -18,6 +18,7 @@
 #include "hw/vfio/vfio-device.h"
 #include "system/iommufd.h"
 #include "hw/s390x/ap-device.h"
+#include "hw/s390x/css.h"
 #include "qemu/error-report.h"
 #include "qemu/event_notifier.h"
 #include "qemu/main-loop.h"
@@ -37,6 +38,7 @@ struct VFIOAPDevice {
     APDevice apdev;
     VFIODevice vdev;
     EventNotifier req_notifier;
+    EventNotifier cfg_notifier;
 };
 
 OBJECT_DECLARE_SIMPLE_TYPE(VFIOAPDevice, VFIO_AP_DEVICE)
@@ -70,6 +72,18 @@ static void vfio_ap_req_notifier_handler(void *opaque)
     }
 }
 
+static void vfio_ap_cfg_chg_notifier_handler(void *opaque)
+{
+    VFIOAPDevice *vapdev = opaque;
+
+    if (!event_notifier_test_and_clear(&vapdev->cfg_notifier)) {
+        return;
+    }
+
+    css_generate_css_crws(0);
+
+}
+
 static bool vfio_ap_register_irq_notifier(VFIOAPDevice *vapdev,
                                           unsigned int irq, Error **errp)
 {
@@ -85,6 +99,10 @@ static bool vfio_ap_register_irq_notifier(VFIOAPDevice *vapdev,
         notifier = &vapdev->req_notifier;
         fd_read = vfio_ap_req_notifier_handler;
         break;
+    case VFIO_AP_CFG_CHG_IRQ_INDEX:
+        notifier = &vapdev->cfg_notifier;
+        fd_read = vfio_ap_cfg_chg_notifier_handler;
+        break;
     default:
         error_setg(errp, "vfio: Unsupported device irq(%d)", irq);
         return false;
@@ -137,6 +155,9 @@ static void vfio_ap_unregister_irq_notifier(VFIOAPDevice *vapdev,
     case VFIO_AP_REQ_IRQ_INDEX:
         notifier = &vapdev->req_notifier;
         break;
+    case VFIO_AP_CFG_CHG_IRQ_INDEX:
+        notifier = &vapdev->cfg_notifier;
+        break;
     default:
         error_report("vfio: Unsupported device irq(%d)", irq);
         return;
@@ -176,6 +197,15 @@ static void vfio_ap_realize(DeviceState *dev, Error **errp)
         warn_report_err(err);
     }
 
+    if (!vfio_ap_register_irq_notifier(vapdev, VFIO_AP_CFG_CHG_IRQ_INDEX, &err))
+    {
+        /*
+         * Report this error, but do not make it a failing condition.
+         * Lack of this IRQ in the host does not prevent normal operation.
+         */
+        warn_report_err(err);
+    }
+
     return;
 
 error:
@@ -188,6 +218,7 @@ static void vfio_ap_unrealize(DeviceState *dev)
     VFIOAPDevice *vapdev = VFIO_AP_DEVICE(dev);
 
     vfio_ap_unregister_irq_notifier(vapdev, VFIO_AP_REQ_IRQ_INDEX);
+    vfio_ap_unregister_irq_notifier(vapdev, VFIO_AP_CFG_CHG_IRQ_INDEX);
     vfio_device_detach(&vapdev->vdev);
     g_free(vapdev->vdev.name);
 }
-- 
2.48.1

[RFC PATCH v13 2/4] hw/vfio/ap: store object indicating AP config changed in a queue

[Rorie Reyes]

Creates an object indicating that an AP configuration change event
has been received and stores it in a queue. These objects will later
be used to store event information for an AP configuration change
when the CHSC instruction is intercepted.

Signed-off-by: Rorie Reyes <rreyes@linux.ibm.com>
Reviewed-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 hw/vfio/ap.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/hw/vfio/ap.c b/hw/vfio/ap.c
index 93c74ebedb..681fd4a4f1 100644
--- a/hw/vfio/ap.c
+++ b/hw/vfio/ap.c
@@ -21,6 +21,7 @@
 #include "hw/s390x/css.h"
 #include "qemu/error-report.h"
 #include "qemu/event_notifier.h"
+#include "qemu/lockable.h"
 #include "qemu/main-loop.h"
 #include "qemu/module.h"
 #include "qemu/option.h"
@@ -41,6 +42,15 @@ struct VFIOAPDevice {
     EventNotifier cfg_notifier;
 };
 
+typedef struct APConfigChgEvent {
+    QTAILQ_ENTRY(APConfigChgEvent) next;
+} APConfigChgEvent;
+
+static QTAILQ_HEAD(, APConfigChgEvent) cfg_chg_events =
+    QTAILQ_HEAD_INITIALIZER(cfg_chg_events);
+
+static QemuMutex cfg_chg_events_lock;
+
 OBJECT_DECLARE_SIMPLE_TYPE(VFIOAPDevice, VFIO_AP_DEVICE)
 
 static void vfio_ap_compute_needs_reset(VFIODevice *vdev)
@@ -74,12 +84,19 @@ static void vfio_ap_req_notifier_handler(void *opaque)
 
 static void vfio_ap_cfg_chg_notifier_handler(void *opaque)
 {
+    APConfigChgEvent *cfg_chg_event;
     VFIOAPDevice *vapdev = opaque;
 
     if (!event_notifier_test_and_clear(&vapdev->cfg_notifier)) {
         return;
     }
 
+    cfg_chg_event = g_new0(APConfigChgEvent, 1);
+
+    WITH_QEMU_LOCK_GUARD(&cfg_chg_events_lock) {
+        QTAILQ_INSERT_TAIL(&cfg_chg_events, cfg_chg_event, next);
+    }
+
     css_generate_css_crws(0);
 
 }
-- 
2.48.1

[RFC PATCH v13 3/4] hw/vfio/ap: Storing event information for an AP configuration change event

[Rorie Reyes]

These functions can be invoked by the function that handles interception
of the CHSC SEI instruction for requests indicating the accessibility of
one or more adjunct processors has changed.

Signed-off-by: Rorie Reyes <rreyes@linux.ibm.com>
Reviewed-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 hw/vfio/ap.c                 | 40 ++++++++++++++++++++++++++++++++++++
 include/hw/s390x/ap-bridge.h | 39 +++++++++++++++++++++++++++++++++++
 2 files changed, 79 insertions(+)

diff --git a/hw/vfio/ap.c b/hw/vfio/ap.c
index 681fd4a4f1..874e0d1eaf 100644
--- a/hw/vfio/ap.c
+++ b/hw/vfio/ap.c
@@ -10,6 +10,7 @@
  * directory.
  */
 
+#include <stdbool.h>
 #include "qemu/osdep.h"
 #include CONFIG_DEVICES /* CONFIG_IOMMUFD */
 #include <linux/vfio.h>
@@ -101,6 +102,38 @@ static void vfio_ap_cfg_chg_notifier_handler(void *opaque)
 
 }
 
+int ap_chsc_sei_nt0_get_event(void *res)
+{
+    ChscSeiNt0Res *nt0_res  = (ChscSeiNt0Res *)res;
+    APConfigChgEvent *cfg_chg_event;
+
+    WITH_QEMU_LOCK_GUARD(&cfg_chg_events_lock) {
+        if (QTAILQ_EMPTY(&cfg_chg_events)) {
+            return EVENT_INFORMATION_NOT_STORED;
+        }
+
+        cfg_chg_event = QTAILQ_FIRST(&cfg_chg_events);
+        QTAILQ_REMOVE(&cfg_chg_events, cfg_chg_event, next);
+    }
+
+    memset(nt0_res, 0, sizeof(*nt0_res));
+    g_free(cfg_chg_event);
+    nt0_res->flags |= PENDING_EVENT_INFO_BITMASK;
+    nt0_res->length = sizeof(ChscSeiNt0Res);
+    nt0_res->code = NT0_RES_RESPONSE_CODE;
+    nt0_res->nt = NT0_RES_NT_DEFAULT;
+    nt0_res->rs = NT0_RES_RS_AP_CHANGE;
+    nt0_res->cc = NT0_RES_CC_AP_CHANGE;
+
+    return EVENT_INFORMATION_STORED;
+}
+
+bool ap_chsc_sei_nt0_have_event(void)
+{
+    QEMU_LOCK_GUARD(&cfg_chg_events_lock);
+    return !QTAILQ_EMPTY(&cfg_chg_events);
+}
+
 static bool vfio_ap_register_irq_notifier(VFIOAPDevice *vapdev,
                                           unsigned int irq, Error **errp)
 {
@@ -197,6 +230,13 @@ static void vfio_ap_realize(DeviceState *dev, Error **errp)
     VFIOAPDevice *vapdev = VFIO_AP_DEVICE(dev);
     VFIODevice *vbasedev = &vapdev->vdev;
 
+    static bool lock_initialized;
+
+    if (!lock_initialized) {
+        qemu_mutex_init(&cfg_chg_events_lock);
+        lock_initialized = true;
+    }
+
     if (!vfio_device_get_name(vbasedev, errp)) {
         return;
     }
diff --git a/include/hw/s390x/ap-bridge.h b/include/hw/s390x/ap-bridge.h
index 470e439a98..7efc52928d 100644
--- a/include/hw/s390x/ap-bridge.h
+++ b/include/hw/s390x/ap-bridge.h
@@ -16,4 +16,43 @@
 
 void s390_init_ap(void);
 
+typedef struct ChscSeiNt0Res {
+    uint16_t length;
+    uint16_t code;
+    uint8_t reserved1;
+    uint16_t reserved2;
+    uint8_t nt;
+#define PENDING_EVENT_INFO_BITMASK 0x80;
+    uint8_t flags;
+    uint8_t reserved3;
+    uint8_t rs;
+    uint8_t cc;
+} QEMU_PACKED ChscSeiNt0Res;
+
+#define NT0_RES_RESPONSE_CODE 1
+#define NT0_RES_NT_DEFAULT    0
+#define NT0_RES_RS_AP_CHANGE  5
+#define NT0_RES_CC_AP_CHANGE  3
+
+#define EVENT_INFORMATION_NOT_STORED 1
+#define EVENT_INFORMATION_STORED     0
+
+/**
+ * ap_chsc_sei_nt0_get_event - Retrieve the next pending AP config
+ * change event
+ * @res: Pointer to a ChscSeiNt0Res struct to be filled with event
+ * data
+ *
+ * This function checks for any pending AP config change events and,
+ * if present, populates the provided response structure with the
+ * appropriate SEI NT0 fields.
+ *
+ * Return:
+ *   EVENT_INFORMATION_STORED - An event was available and written to @res
+ *   EVENT_INFORMATION_NOT_STORED - No event was available
+ */
+int ap_chsc_sei_nt0_get_event(void *res);
+
+bool ap_chsc_sei_nt0_have_event(void);
+
 #endif
-- 
2.48.1

[RFC PATCH v13 4/4] s390: implementing CHSC SEI for AP config change

[Rorie Reyes]

Handle interception of the CHSC SEI instruction for requests
indicating the guest's AP configuration has changed.

If configuring --without-default-devices, hw/s390x/ap-stub.c
was created to handle such circumstance. Also added the
following to hw/s390x/meson.build if CONFIG_VFIO_AP is
false, it will use the stub file.

Signed-off-by: Rorie Reyes <rreyes@linux.ibm.com>
Reviewed-by: Anthony Krowiak <akrowiak@linux.ibm.com>
Reviewed-by: Cédric Le Goater <clg@redhat.com>
---
 MAINTAINERS           |  1 +
 hw/s390x/ap-stub.c    | 21 +++++++++++++++++++++
 hw/s390x/meson.build  |  1 +
 target/s390x/ioinst.c | 11 +++++++++--
 4 files changed, 32 insertions(+), 2 deletions(-)
 create mode 100644 hw/s390x/ap-stub.c

diff --git a/MAINTAINERS b/MAINTAINERS
index aa6763077e..1e84bfeaee 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -112,6 +112,7 @@ F: hw/intc/s390_flic.c
 F: hw/intc/s390_flic_kvm.c
 F: hw/s390x/
 F: hw/vfio/ap.c
+F: hw/s390x/ap-stub.c
 F: hw/vfio/ccw.c
 F: hw/watchdog/wdt_diag288.c
 F: include/hw/s390x/
diff --git a/hw/s390x/ap-stub.c b/hw/s390x/ap-stub.c
new file mode 100644
index 0000000000..001fe5f8b0
--- /dev/null
+++ b/hw/s390x/ap-stub.c
@@ -0,0 +1,21 @@
+/*
+ * VFIO based AP matrix device assignment
+ *
+ * Copyright 2025 IBM Corp.
+ * Author(s): Rorie Reyes <rreyes@linux.ibm.com>
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#include "qemu/osdep.h"
+#include "hw/s390x/ap-bridge.h"
+
+int ap_chsc_sei_nt0_get_event(void *res)
+{
+    return EVENT_INFORMATION_NOT_STORED;
+}
+
+bool ap_chsc_sei_nt0_have_event(void)
+{
+    return false;
+}
diff --git a/hw/s390x/meson.build b/hw/s390x/meson.build
index 3bbebfd817..99cbcbd7d6 100644
--- a/hw/s390x/meson.build
+++ b/hw/s390x/meson.build
@@ -33,6 +33,7 @@ s390x_ss.add(when: 'CONFIG_S390_CCW_VIRTIO', if_true: files(
 ))
 s390x_ss.add(when: 'CONFIG_TERMINAL3270', if_true: files('3270-ccw.c'))
 s390x_ss.add(when: 'CONFIG_VFIO', if_true: files('s390-pci-vfio.c'))
+s390x_ss.add(when: 'CONFIG_VFIO_AP', if_false: files('ap-stub.c'))
 
 virtio_ss = ss.source_set()
 virtio_ss.add(files('virtio-ccw.c'))
diff --git a/target/s390x/ioinst.c b/target/s390x/ioinst.c
index fe62ba5b06..2320dd4c12 100644
--- a/target/s390x/ioinst.c
+++ b/target/s390x/ioinst.c
@@ -18,6 +18,7 @@
 #include "trace.h"
 #include "hw/s390x/s390-pci-bus.h"
 #include "target/s390x/kvm/pv.h"
+#include "hw/s390x/ap-bridge.h"
 
 /* All I/O instructions but chsc use the s format */
 static uint64_t get_address_from_regs(CPUS390XState *env, uint32_t ipb,
@@ -574,13 +575,19 @@ out:
 
 static int chsc_sei_nt0_get_event(void *res)
 {
-    /* no events yet */
+    if (s390_has_feat(S390_FEAT_AP)) {
+        return ap_chsc_sei_nt0_get_event(res);
+    }
+
     return 1;
 }
 
 static int chsc_sei_nt0_have_event(void)
 {
-    /* no events yet */
+    if (s390_has_feat(S390_FEAT_AP)) {
+        return ap_chsc_sei_nt0_have_event();
+    }
+
     return 0;
 }
 
-- 
2.48.1

Re: [RFC PATCH v13 3/4] hw/vfio/ap: Storing event information for an AP configuration change event

[Cédric Le Goater]

On 6/9/25 18:44, Rorie Reyes wrote:
> These functions can be invoked by the function that handles interception
> of the CHSC SEI instruction for requests indicating the accessibility of
> one or more adjunct processors has changed.
> 
> Signed-off-by: Rorie Reyes <rreyes@linux.ibm.com>
> Reviewed-by: Anthony Krowiak <akrowiak@linux.ibm.com>
> ---
>   hw/vfio/ap.c                 | 40 ++++++++++++++++++++++++++++++++++++
>   include/hw/s390x/ap-bridge.h | 39 +++++++++++++++++++++++++++++++++++
>   2 files changed, 79 insertions(+)
> 
> diff --git a/hw/vfio/ap.c b/hw/vfio/ap.c
> index 681fd4a4f1..874e0d1eaf 100644
> --- a/hw/vfio/ap.c
> +++ b/hw/vfio/ap.c
> @@ -10,6 +10,7 @@
>    * directory.
>    */
>   
> +#include <stdbool.h>
>   #include "qemu/osdep.h"
>   #include CONFIG_DEVICES /* CONFIG_IOMMUFD */
>   #include <linux/vfio.h>
> @@ -101,6 +102,38 @@ static void vfio_ap_cfg_chg_notifier_handler(void *opaque)
>   
>   }
>   
> +int ap_chsc_sei_nt0_get_event(void *res)
> +{
> +    ChscSeiNt0Res *nt0_res  = (ChscSeiNt0Res *)res;
> +    APConfigChgEvent *cfg_chg_event;
> +
> +    WITH_QEMU_LOCK_GUARD(&cfg_chg_events_lock) {
> +        if (QTAILQ_EMPTY(&cfg_chg_events)) {
> +            return EVENT_INFORMATION_NOT_STORED;
> +        }
> +
> +        cfg_chg_event = QTAILQ_FIRST(&cfg_chg_events);
> +        QTAILQ_REMOVE(&cfg_chg_events, cfg_chg_event, next);
> +    }
> +
> +    memset(nt0_res, 0, sizeof(*nt0_res));
> +    g_free(cfg_chg_event);
> +    nt0_res->flags |= PENDING_EVENT_INFO_BITMASK;
> +    nt0_res->length = sizeof(ChscSeiNt0Res);
> +    nt0_res->code = NT0_RES_RESPONSE_CODE;
> +    nt0_res->nt = NT0_RES_NT_DEFAULT;
> +    nt0_res->rs = NT0_RES_RS_AP_CHANGE;
> +    nt0_res->cc = NT0_RES_CC_AP_CHANGE;
> +
> +    return EVENT_INFORMATION_STORED;
> +}
> +
> +bool ap_chsc_sei_nt0_have_event(void)
> +{
> +    QEMU_LOCK_GUARD(&cfg_chg_events_lock);
> +    return !QTAILQ_EMPTY(&cfg_chg_events);
> +}
> +
>   static bool vfio_ap_register_irq_notifier(VFIOAPDevice *vapdev,
>                                             unsigned int irq, Error **errp)
>   {
> @@ -197,6 +230,13 @@ static void vfio_ap_realize(DeviceState *dev, Error **errp)
>       VFIOAPDevice *vapdev = VFIO_AP_DEVICE(dev);
>       VFIODevice *vbasedev = &vapdev->vdev;
>   
> +    static bool lock_initialized;
> +
> +    if (!lock_initialized) {
> +        qemu_mutex_init(&cfg_chg_events_lock);
> +        lock_initialized = true;
> +    }
> +

cfg_chg_events_lock could be initialized in a __constructor__ routine.


Thanks,

C.



>       if (!vfio_device_get_name(vbasedev, errp)) {
>           return;
>       }
> diff --git a/include/hw/s390x/ap-bridge.h b/include/hw/s390x/ap-bridge.h
> index 470e439a98..7efc52928d 100644
> --- a/include/hw/s390x/ap-bridge.h
> +++ b/include/hw/s390x/ap-bridge.h
> @@ -16,4 +16,43 @@
>   
>   void s390_init_ap(void);
>   
> +typedef struct ChscSeiNt0Res {
> +    uint16_t length;
> +    uint16_t code;
> +    uint8_t reserved1;
> +    uint16_t reserved2;
> +    uint8_t nt;
> +#define PENDING_EVENT_INFO_BITMASK 0x80;
> +    uint8_t flags;
> +    uint8_t reserved3;
> +    uint8_t rs;
> +    uint8_t cc;
> +} QEMU_PACKED ChscSeiNt0Res;
> +
> +#define NT0_RES_RESPONSE_CODE 1
> +#define NT0_RES_NT_DEFAULT    0
> +#define NT0_RES_RS_AP_CHANGE  5
> +#define NT0_RES_CC_AP_CHANGE  3
> +
> +#define EVENT_INFORMATION_NOT_STORED 1
> +#define EVENT_INFORMATION_STORED     0
> +
> +/**
> + * ap_chsc_sei_nt0_get_event - Retrieve the next pending AP config
> + * change event
> + * @res: Pointer to a ChscSeiNt0Res struct to be filled with event
> + * data
> + *
> + * This function checks for any pending AP config change events and,
> + * if present, populates the provided response structure with the
> + * appropriate SEI NT0 fields.
> + *
> + * Return:
> + *   EVENT_INFORMATION_STORED - An event was available and written to @res
> + *   EVENT_INFORMATION_NOT_STORED - No event was available
> + */
> +int ap_chsc_sei_nt0_get_event(void *res);
> +
> +bool ap_chsc_sei_nt0_have_event(void);
> +
>   #endif

Re: [RFC PATCH v13 0/4] Report vfio-ap configuration changes

[Cédric Le Goater]

On 6/9/25 18:44, Rorie Reyes wrote:
> Changelog:
> v13:
> - added lock to 'vfio_ap_cfg_chg_notifier_handler' in patch 2
> - added RBs for patch 2 and 3 from Tony
> 
> v12:
> - adding locks to 'ap_chsc_sei_nt0_have_event' and 'ap_chsc_sei_nt0_get_event'
> 
> v11:
> - reverted return type to int for 'ap_chsc_sei_nt0_get_event'
>     - files reflected are 'ap.c', 'ap-bridge'h, and 'ap-stub.c'
> - using defined variables to represent return 0 and 1 to reflect logical sense
> - update documentation for 'ap_chsc_sei_nt0_get_event' to reflect return types
> 
> v10:
> - Added reviewed by for patch 4 by Tony and Cedric
>     - review needed for patch 2 and 3
> - changed 'cfg_chg_events' to a static variable
> - locked 'cfg_chg_events' using QemuMutex
> - removed ';' at the end of the definition for NT0
> - returning bools for 'ap_chsc_sei_nt0_get_event' and
> 'ap_chsc_sei_nt0_have_event'
> - updated the header file that contains 'ap_chsc_sei_nt0_get_event' and
> 'ap_chsc_sei_nt0_have_event' to a bool function
> - added documentation explaining the returning bool functions
> - whitespace clean up
> 
> v9:
> - added SPDX licensing to newly created file 'hw/s390x/ap-stub.c'
> 
> v8:
> - fixed windows cross-compile build
> - moved /hw/vfio/ap-stub.c to /hw/s390x/ap-stub.c
> - updated the use of stub file to MAINTAINERS to reflect new location
> - removed if_false for 'CONFIG_VFIO_AP' statement from /hw/vfio/meson.build
> - added if_false for 'CONFIG_VFIO_AP' to use ap-stub.c in /hw/s390x/meson.build
> - all those changes still address '--without-default-devices' issue from v5
> 
> v7:
> - Dropped initial commit for linux-header file vfio.h since I created two new commits
> to address the changes made in v6
> - Moved patches 6 and 7 to the beginning of the series after dropping the first patch
>     - Because I dropped the initial commit for linux-header file vfio.h, I had to add
> VFIO_AP_CFG_CHG_IRQ_INDEX
> - Resyncing latest to v6.15-rc3
> - Still need Thomas Huth's review of v5 changes for patch 6/6
> 
> v6:
> - Updating the update-linux-headers script to address kernel commit change 8a14
> - Update headers to retrieve uapi information for vfio-ap for update to Linux v6.15-rc1
> - Still need Thomas Huth's review of v5 changes for patch 7/7 (see below)
> 
> v5:
> - configuring using the '--without-default-devices' fails when building the code
> - created a stub file for functions ap_chsc_sei_nt0_get_event and ap_chsc_sei_nt0_have_event
> - add if_false for 'CONFIG_VFIO_AP' use ap-stub.c in meson.build
> - add the use of the stub file to MAINTAINERS since it's a new file
> 
> v4:
> - allocating cfg_chg_event before inserting into the queue
> - calling nt0_have_event in if loop to check if there are any
> elemenets in the queue, then calling QTAILQ_FIRST when the check
> passes
> - moving memset() after the check
> 
> v3:
> - changes that were made to patch 3/5 should have been made in
> patch 2/5
> 
> v2:
> - removed warnings that weren't needed
> - added unregister function
> - removed whitelines
> - changed variable names for consistency
> - removed rc variable and returning 1 or 0 outright
> - reversed logics for if statements
> - using g_free() instead of free()
> - replaced hardcoded numeric values by defining them with #define
> in the header
> 
> --------------------------------------------------------------------------
> 
> Rorie Reyes (4):
>    hw/vfio/ap: notification handler for AP config changed event
>    hw/vfio/ap: store object indicating AP config changed in a queue
>    hw/vfio/ap: Storing event information for an AP configuration change
>      event
>    s390: implementing CHSC SEI for AP config change
> 
>   MAINTAINERS                  |  1 +
>   hw/s390x/ap-stub.c           | 21 +++++++++
>   hw/s390x/meson.build         |  1 +
>   hw/vfio/ap.c                 | 88 ++++++++++++++++++++++++++++++++++++
>   include/hw/s390x/ap-bridge.h | 39 ++++++++++++++++
>   target/s390x/ioinst.c        | 11 ++++-
>   6 files changed, 159 insertions(+), 2 deletions(-)
>   create mode 100644 hw/s390x/ap-stub.c
> 


Applied to vfio-next.

Thanks,

C.

Re: [RFC PATCH v13 4/4] s390: implementing CHSC SEI for AP config change

[Matthew Rosato]

On 6/9/25 12:44 PM, Rorie Reyes wrote:
> Handle interception of the CHSC SEI instruction for requests
> indicating the guest's AP configuration has changed.
> 
> If configuring --without-default-devices, hw/s390x/ap-stub.c
> was created to handle such circumstance. Also added the
> following to hw/s390x/meson.build if CONFIG_VFIO_AP is
> false, it will use the stub file.
> 
> Signed-off-by: Rorie Reyes <rreyes@linux.ibm.com>
> Reviewed-by: Anthony Krowiak <akrowiak@linux.ibm.com>
> Reviewed-by: Cédric Le Goater <clg@redhat.com>

FYI, this patch (or some part of this series) breaks hotplug for PCI devices on s390x.  I verified it breaks with both PCI passthrough and virtio-pci via virsh attach-device.

I get greeted with the following message:
qemu-system-s390x: ../util/qemu-thread-posix.c:92: qemu_mutex_lock_impl: Assertion `mutex->initialized' failed.
2025-06-26 14:12:23.031+0000: shutting down, reason=crashed

As a quick test, reverting this patch resolves the issue.  

Another test shows that if I have a vfio-ap device attached to my guest before I try the zPCI hotplug, everything works fine (because vfio_ap_realize was driven -- see more info below).

I note in the failing call chain that in response to the hotplug event for a zPCI device we end up calling into hw/vfio/ap.c and getting some locks which seems wrong:

(gdb) bt
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x000003ffa85ad636 in __pthread_kill_internal (threadid=<optimized out>, signo=6) at pthread_kill.c:78
#2  0x000003ffa8553c20 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x000003ffa8533e3c in __GI_abort () at abort.c:79
#4  0x000003ffa854ac9a in __assert_fail_base
    (fmt=<optimized out>, assertion=assertion@entry=0x2aa3fa76414 "mutex->initialized", file=file@entry=0x2aa3fa763f8 "../util/qemu-thread-posix.c", line=line@entry=92, function=function@entry=0x2aa3fa7647a <__PRETTY_FUNCTION__.23> "qemu_mutex_lock_impl") at assert.c:94
#5  0x000003ffa854acf4 in __assert_fail
    (assertion=0x2aa3fa76414 "mutex->initialized", file=0x2aa3fa763f8 "../util/qemu-thread-posix.c", line=92, function=0x2aa3fa7647a <__PRETTY_FUNCTION__.23> "qemu_mutex_lock_impl") at assert.c:103
#6  0x000002aa3f83ef02 in qemu_mutex_lock_impl (mutex=0x2aa3fed9e08 <cfg_chg_events_lock>, file=0x2aa3f9e653e "/usr/src/qemu/include/qemu/lockable.h", line=56) at ../util/qemu-thread-posix.c:92
#7  0x000002aa3f4c3a70 in qemu_lockable_mutex_lock (x=0x2aa3fed9e08 <cfg_chg_events_lock>) at /usr/src/qemu/include/qemu/lockable.h:56
#8  0x000002aa3f4c3b00 in qemu_lockable_lock (x=0x3fd076f7730) at /usr/src/qemu/include/qemu/lockable.h:110
#9  0x000002aa3f4c3b82 in qemu_lockable_auto_lock (x=0x3fd076f7730) at /usr/src/qemu/include/qemu/lockable.h:120
#10 0x000002aa3f4c3f90 in ap_chsc_sei_nt0_get_event (res=0x3fd076f79b8) at ../hw/vfio/ap.c:110
^^^^^^^^
#11 0x000002aa3f3a59ea in chsc_sei_nt0_get_event (res=0x3fd076f79b8) at ../target/s390x/ioinst.c:579
#12 0x000002aa3f3a5b76 in ioinst_handle_chsc_sei (req=0x3fd076f79a8, res=0x3fd076f79b8) at ../target/s390x/ioinst.c:620
#13 0x000002aa3f3a6062 in ioinst_handle_chsc (cpu=0x2aa40cc9430, ipb=2097152, ra=0) at ../target/s390x/ioinst.c:708
#14 0x000002aa3f4647d2 in handle_b2 (cpu=0x2aa40cc9430, run=0x3ffa4180000, ipa1=95 '_') at ../target/s390x/kvm/kvm.c:1218
#15 0x000002aa3f465db2 in handle_instruction (cpu=0x2aa40cc9430, run=0x3ffa4180000) at ../target/s390x/kvm/kvm.c:1627
#16 0x000002aa3f4661da in handle_intercept (cpu=0x2aa40cc9430) at ../target/s390x/kvm/kvm.c:1709
#17 0x000002aa3f466c10 in kvm_arch_handle_exit (cs=0x2aa40cc9430, run=0x3ffa4180000) at ../target/s390x/kvm/kvm.c:1914
#18 0x000002aa3f523c98 in kvm_cpu_exec (cpu=0x2aa40cc9430) at ../accel/kvm/kvm-all.c:3294
#19 0x000002aa3f527f52 in kvm_vcpu_thread_fn (arg=0x2aa40cc9430) at ../accel/kvm/kvm-accel-ops.c:51
#20 0x000002aa3f840002 in qemu_thread_start (args=0x2aa40ccd090) at ../util/qemu-thread-posix.c:393
#21 0x000003ffa85ab622 in start_thread (arg=0x3fd076fe8c0) at pthread_create.c:448
#22 0x000003ffa862b390 in thread_start () at ../sysdeps/unix/sysv/linux/s390/s390-64/clone3.S:71


Note that PCI devices were already using SEI before this series (e.g. see hw/s390x/s390-pci-bus.c:pci_chsc_sei_nt2_get_event()); I think the problem here is that nt0 events are looked for implicitly in the shared handler (ioinst_handle_chsc_sei()) which is fine -- but you cannot go acquiring locks from ap.c that might have never been initialized (cfg_chg_events_lock which is only initialized in vfio_ap_realize()); that's why we ultimately crash.

AFAICT this portion of the handler including mutex should be moved out of ap.c into chsc_sei_nt0_get_event().  When vfio_ap is not in use, we can't call into it.  Can't you also build without VFIO_AP?  I didn't try it but that sure seems like it would cause issues too.  

I guess the mutex and list would then need to be part of something common to s390x like S390CcwMachineState.

Alternatively, you could leave those in vfio_ap but would need some kind of indicator that vfio_ap is "armed" for these kinds of events and only bother calling into vfio_ap to look at cfg_chg_events when that indicator has been set (e.g. vfio_ap_realize has been called once.  And it actually looks like you already do something like this via lock_initialized in ap.c). 
Otherwise, we should always return EVENT_INFORMATION_NOT_STORED like the old code did (via return 1).

Or maybe someone else has a better idea.


> ---
>  MAINTAINERS           |  1 +
>  hw/s390x/ap-stub.c    | 21 +++++++++++++++++++++
>  hw/s390x/meson.build  |  1 +
>  target/s390x/ioinst.c | 11 +++++++++--
>  4 files changed, 32 insertions(+), 2 deletions(-)
>  create mode 100644 hw/s390x/ap-stub.c
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index aa6763077e..1e84bfeaee 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -112,6 +112,7 @@ F: hw/intc/s390_flic.c
>  F: hw/intc/s390_flic_kvm.c
>  F: hw/s390x/
>  F: hw/vfio/ap.c
> +F: hw/s390x/ap-stub.c
>  F: hw/vfio/ccw.c
>  F: hw/watchdog/wdt_diag288.c
>  F: include/hw/s390x/
> diff --git a/hw/s390x/ap-stub.c b/hw/s390x/ap-stub.c
> new file mode 100644
> index 0000000000..001fe5f8b0
> --- /dev/null
> +++ b/hw/s390x/ap-stub.c
> @@ -0,0 +1,21 @@
> +/*
> + * VFIO based AP matrix device assignment
> + *
> + * Copyright 2025 IBM Corp.
> + * Author(s): Rorie Reyes <rreyes@linux.ibm.com>
> + *
> + * SPDX-License-Identifier: GPL-2.0-or-later
> + */
> +
> +#include "qemu/osdep.h"
> +#include "hw/s390x/ap-bridge.h"
> +
> +int ap_chsc_sei_nt0_get_event(void *res)
> +{
> +    return EVENT_INFORMATION_NOT_STORED;
> +}
> +
> +bool ap_chsc_sei_nt0_have_event(void)
> +{
> +    return false;
> +}
> diff --git a/hw/s390x/meson.build b/hw/s390x/meson.build
> index 3bbebfd817..99cbcbd7d6 100644
> --- a/hw/s390x/meson.build
> +++ b/hw/s390x/meson.build
> @@ -33,6 +33,7 @@ s390x_ss.add(when: 'CONFIG_S390_CCW_VIRTIO', if_true: files(
>  ))
>  s390x_ss.add(when: 'CONFIG_TERMINAL3270', if_true: files('3270-ccw.c'))
>  s390x_ss.add(when: 'CONFIG_VFIO', if_true: files('s390-pci-vfio.c'))
> +s390x_ss.add(when: 'CONFIG_VFIO_AP', if_false: files('ap-stub.c'))
>  
>  virtio_ss = ss.source_set()
>  virtio_ss.add(files('virtio-ccw.c'))
> diff --git a/target/s390x/ioinst.c b/target/s390x/ioinst.c
> index fe62ba5b06..2320dd4c12 100644
> --- a/target/s390x/ioinst.c
> +++ b/target/s390x/ioinst.c
> @@ -18,6 +18,7 @@
>  #include "trace.h"
>  #include "hw/s390x/s390-pci-bus.h"
>  #include "target/s390x/kvm/pv.h"
> +#include "hw/s390x/ap-bridge.h"
>  
>  /* All I/O instructions but chsc use the s format */
>  static uint64_t get_address_from_regs(CPUS390XState *env, uint32_t ipb,
> @@ -574,13 +575,19 @@ out:
>  
>  static int chsc_sei_nt0_get_event(void *res)
>  {
> -    /* no events yet */
> +    if (s390_has_feat(S390_FEAT_AP)) {
> +        return ap_chsc_sei_nt0_get_event(res);
> +    }
> +
>      return 1;
>  }
>  
>  static int chsc_sei_nt0_have_event(void)
>  {
> -    /* no events yet */
> +    if (s390_has_feat(S390_FEAT_AP)) {
> +        return ap_chsc_sei_nt0_have_event();
> +    }
> +
>      return 0;
>  }
>  

Re: [RFC PATCH v13 4/4] s390: implementing CHSC SEI for AP config change

[Anthony Krowiak]

On 6/26/25 12:19 PM, Matthew Rosato wrote:
> On 6/9/25 12:44 PM, Rorie Reyes wrote:
>> Handle interception of the CHSC SEI instruction for requests
>> indicating the guest's AP configuration has changed.
>>
>> If configuring --without-default-devices, hw/s390x/ap-stub.c
>> was created to handle such circumstance. Also added the
>> following to hw/s390x/meson.build if CONFIG_VFIO_AP is
>> false, it will use the stub file.
>>
>> Signed-off-by: Rorie Reyes <rreyes@linux.ibm.com>
>> Reviewed-by: Anthony Krowiak <akrowiak@linux.ibm.com>
>> Reviewed-by: Cédric Le Goater <clg@redhat.com>
> FYI, this patch (or some part of this series) breaks hotplug for PCI devices on s390x.  I verified it breaks with both PCI passthrough and virtio-pci via virsh attach-device.
>
> I get greeted with the following message:
> qemu-system-s390x: ../util/qemu-thread-posix.c:92: qemu_mutex_lock_impl: Assertion `mutex->initialized' failed.
> 2025-06-26 14:12:23.031+0000: shutting down, reason=crashed
>
> As a quick test, reverting this patch resolves the issue.
>
> Another test shows that if I have a vfio-ap device attached to my guest before I try the zPCI hotplug, everything works fine (because vfio_ap_realize was driven -- see more info below).
>
> I note in the failing call chain that in response to the hotplug event for a zPCI device we end up calling into hw/vfio/ap.c and getting some locks which seems wrong:
>
> (gdb) bt
> #0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
> #1  0x000003ffa85ad636 in __pthread_kill_internal (threadid=<optimized out>, signo=6) at pthread_kill.c:78
> #2  0x000003ffa8553c20 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
> #3  0x000003ffa8533e3c in __GI_abort () at abort.c:79
> #4  0x000003ffa854ac9a in __assert_fail_base
>      (fmt=<optimized out>, assertion=assertion@entry=0x2aa3fa76414 "mutex->initialized", file=file@entry=0x2aa3fa763f8 "../util/qemu-thread-posix.c", line=line@entry=92, function=function@entry=0x2aa3fa7647a <__PRETTY_FUNCTION__.23> "qemu_mutex_lock_impl") at assert.c:94
> #5  0x000003ffa854acf4 in __assert_fail
>      (assertion=0x2aa3fa76414 "mutex->initialized", file=0x2aa3fa763f8 "../util/qemu-thread-posix.c", line=92, function=0x2aa3fa7647a <__PRETTY_FUNCTION__.23> "qemu_mutex_lock_impl") at assert.c:103
> #6  0x000002aa3f83ef02 in qemu_mutex_lock_impl (mutex=0x2aa3fed9e08 <cfg_chg_events_lock>, file=0x2aa3f9e653e "/usr/src/qemu/include/qemu/lockable.h", line=56) at ../util/qemu-thread-posix.c:92
> #7  0x000002aa3f4c3a70 in qemu_lockable_mutex_lock (x=0x2aa3fed9e08 <cfg_chg_events_lock>) at /usr/src/qemu/include/qemu/lockable.h:56
> #8  0x000002aa3f4c3b00 in qemu_lockable_lock (x=0x3fd076f7730) at /usr/src/qemu/include/qemu/lockable.h:110
> #9  0x000002aa3f4c3b82 in qemu_lockable_auto_lock (x=0x3fd076f7730) at /usr/src/qemu/include/qemu/lockable.h:120
> #10 0x000002aa3f4c3f90 in ap_chsc_sei_nt0_get_event (res=0x3fd076f79b8) at ../hw/vfio/ap.c:110
> ^^^^^^^^
> #11 0x000002aa3f3a59ea in chsc_sei_nt0_get_event (res=0x3fd076f79b8) at ../target/s390x/ioinst.c:579
> #12 0x000002aa3f3a5b76 in ioinst_handle_chsc_sei (req=0x3fd076f79a8, res=0x3fd076f79b8) at ../target/s390x/ioinst.c:620
> #13 0x000002aa3f3a6062 in ioinst_handle_chsc (cpu=0x2aa40cc9430, ipb=2097152, ra=0) at ../target/s390x/ioinst.c:708
> #14 0x000002aa3f4647d2 in handle_b2 (cpu=0x2aa40cc9430, run=0x3ffa4180000, ipa1=95 '_') at ../target/s390x/kvm/kvm.c:1218
> #15 0x000002aa3f465db2 in handle_instruction (cpu=0x2aa40cc9430, run=0x3ffa4180000) at ../target/s390x/kvm/kvm.c:1627
> #16 0x000002aa3f4661da in handle_intercept (cpu=0x2aa40cc9430) at ../target/s390x/kvm/kvm.c:1709
> #17 0x000002aa3f466c10 in kvm_arch_handle_exit (cs=0x2aa40cc9430, run=0x3ffa4180000) at ../target/s390x/kvm/kvm.c:1914
> #18 0x000002aa3f523c98 in kvm_cpu_exec (cpu=0x2aa40cc9430) at ../accel/kvm/kvm-all.c:3294
> #19 0x000002aa3f527f52 in kvm_vcpu_thread_fn (arg=0x2aa40cc9430) at ../accel/kvm/kvm-accel-ops.c:51
> #20 0x000002aa3f840002 in qemu_thread_start (args=0x2aa40ccd090) at ../util/qemu-thread-posix.c:393
> #21 0x000003ffa85ab622 in start_thread (arg=0x3fd076fe8c0) at pthread_create.c:448
> #22 0x000003ffa862b390 in thread_start () at ../sysdeps/unix/sysv/linux/s390/s390-64/clone3.S:71
>
>
> Note that PCI devices were already using SEI before this series (e.g. see hw/s390x/s390-pci-bus.c:pci_chsc_sei_nt2_get_event()); I think the problem here is that nt0 events are looked for implicitly in the shared handler (ioinst_handle_chsc_sei()) which is fine -- but you cannot go acquiring locks from ap.c that might have never been initialized (cfg_chg_events_lock which is only initialized in vfio_ap_realize()); that's why we ultimately crash.

The following check is done in both the chsc_sei_nt0_get_event and 
chsc_sei_nt0_have_event
functions before calling into the AP versions of those functions:

     if (s390_has_feat(S390_FEAT_AP)) {
         return ap_chsc_sei_nt0_get_event(res);
     }

Given that the call to AP is made, it looks like AP is installed on the 
guest. So it seems the reason
that the vfio_ap_realize() function does not get called is because a 
mediated device is not
passed through to the guest. This check alone, therefore, does not 
protect us against making
a call into the ap_chsc_sei_nt0_get/have_event() functions which results 
in the problem you
are seeing.

>
> AFAICT this portion of the handler including mutex should be moved out of ap.c into chsc_sei_nt0_get_event().  When vfio_ap is not in use, we can't call into it.  Can't you also build without VFIO_AP?  I didn't try it but that sure seems like it would cause issues too.

If that were the case, I think the checks I mentioned above would be 
sufficient because
the S390_FEAT_AP feature could not be set.

>    
>
> I guess the mutex and list would then need to be part of something common to s390x like S390CcwMachineState.
>
> Alternatively, you could leave those in vfio_ap but would need some kind of indicator that vfio_ap is "armed" for these kinds of events and only bother calling into vfio_ap to look at cfg_chg_events when that indicator has been set (e.g. vfio_ap_realize has been called once.  And it actually looks like you already do something like this via lock_initialized in ap.c).
> Otherwise, we should always return EVENT_INFORMATION_NOT_STORED like the old code did (via return 1).
>
> Or maybe someone else has a better idea.

What about moving lock initialization to the vfio_ap_class_init() function?

>
>
>> ---
>>   MAINTAINERS           |  1 +
>>   hw/s390x/ap-stub.c    | 21 +++++++++++++++++++++
>>   hw/s390x/meson.build  |  1 +
>>   target/s390x/ioinst.c | 11 +++++++++--
>>   4 files changed, 32 insertions(+), 2 deletions(-)
>>   create mode 100644 hw/s390x/ap-stub.c
>>
>> diff --git a/MAINTAINERS b/MAINTAINERS
>> index aa6763077e..1e84bfeaee 100644
>> --- a/MAINTAINERS
>> +++ b/MAINTAINERS
>> @@ -112,6 +112,7 @@ F: hw/intc/s390_flic.c
>>   F: hw/intc/s390_flic_kvm.c
>>   F: hw/s390x/
>>   F: hw/vfio/ap.c
>> +F: hw/s390x/ap-stub.c
>>   F: hw/vfio/ccw.c
>>   F: hw/watchdog/wdt_diag288.c
>>   F: include/hw/s390x/
>> diff --git a/hw/s390x/ap-stub.c b/hw/s390x/ap-stub.c
>> new file mode 100644
>> index 0000000000..001fe5f8b0
>> --- /dev/null
>> +++ b/hw/s390x/ap-stub.c
>> @@ -0,0 +1,21 @@
>> +/*
>> + * VFIO based AP matrix device assignment
>> + *
>> + * Copyright 2025 IBM Corp.
>> + * Author(s): Rorie Reyes <rreyes@linux.ibm.com>
>> + *
>> + * SPDX-License-Identifier: GPL-2.0-or-later
>> + */
>> +
>> +#include "qemu/osdep.h"
>> +#include "hw/s390x/ap-bridge.h"
>> +
>> +int ap_chsc_sei_nt0_get_event(void *res)
>> +{
>> +    return EVENT_INFORMATION_NOT_STORED;
>> +}
>> +
>> +bool ap_chsc_sei_nt0_have_event(void)
>> +{
>> +    return false;
>> +}
>> diff --git a/hw/s390x/meson.build b/hw/s390x/meson.build
>> index 3bbebfd817..99cbcbd7d6 100644
>> --- a/hw/s390x/meson.build
>> +++ b/hw/s390x/meson.build
>> @@ -33,6 +33,7 @@ s390x_ss.add(when: 'CONFIG_S390_CCW_VIRTIO', if_true: files(
>>   ))
>>   s390x_ss.add(when: 'CONFIG_TERMINAL3270', if_true: files('3270-ccw.c'))
>>   s390x_ss.add(when: 'CONFIG_VFIO', if_true: files('s390-pci-vfio.c'))
>> +s390x_ss.add(when: 'CONFIG_VFIO_AP', if_false: files('ap-stub.c'))
>>   
>>   virtio_ss = ss.source_set()
>>   virtio_ss.add(files('virtio-ccw.c'))
>> diff --git a/target/s390x/ioinst.c b/target/s390x/ioinst.c
>> index fe62ba5b06..2320dd4c12 100644
>> --- a/target/s390x/ioinst.c
>> +++ b/target/s390x/ioinst.c
>> @@ -18,6 +18,7 @@
>>   #include "trace.h"
>>   #include "hw/s390x/s390-pci-bus.h"
>>   #include "target/s390x/kvm/pv.h"
>> +#include "hw/s390x/ap-bridge.h"
>>   
>>   /* All I/O instructions but chsc use the s format */
>>   static uint64_t get_address_from_regs(CPUS390XState *env, uint32_t ipb,
>> @@ -574,13 +575,19 @@ out:
>>   
>>   static int chsc_sei_nt0_get_event(void *res)
>>   {
>> -    /* no events yet */
>> +    if (s390_has_feat(S390_FEAT_AP)) {
>> +        return ap_chsc_sei_nt0_get_event(res);
>> +    }
>> +
>>       return 1;
>>   }
>>   
>>   static int chsc_sei_nt0_have_event(void)
>>   {
>> -    /* no events yet */
>> +    if (s390_has_feat(S390_FEAT_AP)) {
>> +        return ap_chsc_sei_nt0_have_event();
>> +    }
>> +
>>       return 0;
>>   }
>>   

Re: [RFC PATCH v13 4/4] s390: implementing CHSC SEI for AP config change

[Matthew Rosato]

>>
>> AFAICT this portion of the handler including mutex should be moved out of ap.c into chsc_sei_nt0_get_event().  When vfio_ap is not in use, we can't call into it.  Can't you also build without VFIO_AP?  I didn't try it but that sure seems like it would cause issues too.
> 
> If that were the case, I think the checks I mentioned above would be sufficient because
> the S390_FEAT_AP feature could not be set.
> 

OK, I tested it quick (remove VFIO_AP from config) and that appears to be the case; this also restores zPCI hotplug capability since we won't call info hw/vfio/ap.c/.

> 
> What about moving lock initialization to the vfio_ap_class_init() function?
> 

I tried a quick hack of that too and it appears to work, at least for the zPCI regression (note: I did not test the AP portion).

This approach is probably not the way I would have designed it initially...  But it does seem to be the simplest thing to do from a fix perspective.  

Can you or Rorie prepare a proper patch w/ a fixes tag and send (please add me to CC). Also please test the change against the vfio-ap environment; you can also test zPCI but I would also like to run some tests.

Re: [RFC PATCH v13 4/4] s390: implementing CHSC SEI for AP config change

[Rorie Reyes]

On 6/27/25 11:12 AM, Matthew Rosato wrote:
>>> AFAICT this portion of the handler including mutex should be moved out of ap.c into chsc_sei_nt0_get_event().  When vfio_ap is not in use, we can't call into it.  Can't you also build without VFIO_AP?  I didn't try it but that sure seems like it would cause issues too.
>> If that were the case, I think the checks I mentioned above would be sufficient because
>> the S390_FEAT_AP feature could not be set.
>>
> OK, I tested it quick (remove VFIO_AP from config) and that appears to be the case; this also restores zPCI hotplug capability since we won't call info hw/vfio/ap.c/.
>
>> What about moving lock initialization to the vfio_ap_class_init() function?
>>
> I tried a quick hack of that too and it appears to work, at least for the zPCI regression (note: I did not test the AP portion).
>
> This approach is probably not the way I would have designed it initially...  But it does seem to be the simplest thing to do from a fix perspective.
>
> Can you or Rorie prepare a proper patch w/ a fixes tag and send (please add me to CC). Also please test the change against the vfio-ap environment; you can also test zPCI but I would also like to run some tests.
I can make those changes

Re: [RFC PATCH v13 4/4] s390: implementing CHSC SEI for AP config change

[Anthony Krowiak]

Sure

On 6/27/25 11:12 AM, Matthew Rosato wrote:
>>> AFAICT this portion of the handler including mutex should be moved out of ap.c into chsc_sei_nt0_get_event().  When vfio_ap is not in use, we can't call into it.  Can't you also build without VFIO_AP?  I didn't try it but that sure seems like it would cause issues too.
>> If that were the case, I think the checks I mentioned above would be sufficient because
>> the S390_FEAT_AP feature could not be set.
>>
> OK, I tested it quick (remove VFIO_AP from config) and that appears to be the case; this also restores zPCI hotplug capability since we won't call info hw/vfio/ap.c/.
>
>> What about moving lock initialization to the vfio_ap_class_init() function?
>>
> I tried a quick hack of that too and it appears to work, at least for the zPCI regression (note: I did not test the AP portion).
>
> This approach is probably not the way I would have designed it initially...  But it does seem to be the simplest thing to do from a fix perspective.
>
> Can you or Rorie prepare a proper patch w/ a fixes tag and send (please add me to CC). Also please test the change against the vfio-ap environment; you can also test zPCI but I would also like to run some tests.

Re: [RFC PATCH v13 4/4] s390: implementing CHSC SEI for AP config change

[Matthew Rosato]

On 6/26/25 12:19 PM, Matthew Rosato wrote:
> On 6/9/25 12:44 PM, Rorie Reyes wrote:
>> Handle interception of the CHSC SEI instruction for requests
>> indicating the guest's AP configuration has changed.
>>
>> If configuring --without-default-devices, hw/s390x/ap-stub.c
>> was created to handle such circumstance. Also added the
>> following to hw/s390x/meson.build if CONFIG_VFIO_AP is
>> false, it will use the stub file.
>>
>> Signed-off-by: Rorie Reyes <rreyes@linux.ibm.com>
>> Reviewed-by: Anthony Krowiak <akrowiak@linux.ibm.com>
>> Reviewed-by: Cédric Le Goater <clg@redhat.com>
> 
> FYI, this patch (or some part of this series) breaks hotplug for PCI devices on s390x.  I verified it breaks with both PCI passthrough and virtio-pci via virsh attach-device.
> 

I was unaware of 639ff87a1a ("attribute constructor for cfg_chg_events_lock") which apparently was already sitting in vfio-next and merged to master the day after I reported this issue.

FWIW, 639ff87a1a resolves the issue by ensuring the mutex_init happens even if there is no vfio-ap device attached so long as vfio_ap is configured/built (and I like the use of the constructor).

I also now see the stub functions in hw/s390x/ap-stub.c that will be called if vfio_ap is not configured.

So: my concerns are resolved.  Thanks!