[PATCH 0/8] Force detach PCI devices for ACPI-based and PCIe native hot-unplug

[PATCH 0/8] Force detach PCI devices for ACPI-based and PCIe native hot-unplug

[Dongli Zhang]

Hot-unplugging a PCI device can require cooperation from the guest. For
ACPI PCI hotplug, QEMU notifies the guest through ACPI and the guest
eventually writes the ACPI PCI eject register. For PCIe native hotplug,
QEMU notifies the guest through the PCIe hotplug mechanism and waits for
the slot unplug flow to complete. Only after that completion does QEMU
unrealize the device and emit DEVICE_DELETED.

This can leave a device stuck in the unplug pending state when the guest
does not cooperate. Examples include:

1. The guest has panicked, or the relevant ACPI/PCI hotplug driver is
unavailable.

2. The guest is stalled and cannot handle the hot-unplug event. For
example, stalling the Linux [irq/9-acpi] kernel thread can reproduce this
for ACPI-based hot-unplug.

3. The device was attached to a slot that the guest cannot use. For
example, a pcie-root-port only supports slot 0. If a device is added to a
non-zero slot below a pcie-root-port, the guest may never discover the
device and therefore may never complete the unplug request.

The non-zero slot case has also been discussed in:

hw/pci: warn when PCIe device is plugged into non-zero slot of downstream port
https://gitlab.com/qemu-project/qemu/-/commit/ca92eb5defcf9d1c2106341744a73a03cf26e824

hw/pci: add comment to explain checking for available function 0 in pci hotplug
https://gitlab.com/qemu-project/qemu/-/commit/67d045a0ef5b9c5f871c3a1d87325a8a42d2b9d5

pci: don't skip function 0 occupancy verification for devfn auto assign
https://gitlab.com/qemu-project/qemu/-/commit/e228d62b4af29bca698ec57efdceb46f392f5444

For example, if root-port.1 is a pcie-root-port, the following command adds
a vhost-scsi-pci device to an invalid slot:

(qemu) device_add vhost-scsi-pci,id=scsi01,wwpn=naa.5001405324af0985,bus=root-port.1,addr=01.0
warning: PCI: slot 1 is not valid for vhost-scsi-pci, parent device only allows plugging into slot 0.

In this situation, the device may be impossible to remove through normal
guest-cooperative hot-unplug. Production environments also need a host side
recovery option when the guest kernel is the reason that unplug does not
complete.

This series adds a force option to QMP device_del and HMP device_del. When
requested, QEMU asks the selected hotplug controller to complete the unplug
through a new force_unplug callback.

This series implements forced unplug for ACPI PCI hotplug and PCIe native
hotplug, which cover common pc/q35/virt cases. SHPC is not implemented by
this series.


Dongli Zhang (8):
  qdev: Add force argument to qdev_unplug
  qdev: hotplug: Add force_unplug handler callback
  qdev: Support forced device_del in QMP and HMP
  hw/acpi/pcihp: acpi/pcihp: Add forced slot unplug helper
  hw/acpi/piix4: Support forced PCI unplug
  hw/acpi/ich9: Support forced PCI unplug
  hw/acpi/ged: Support forced PCI unplug
  hw/pci/pcie: Support forced PCIe native unplug

 hmp-commands.hx                 | 11 ++++++-----
 hw/acpi/acpi-pci-hotplug-stub.c |  6 ++++++
 hw/acpi/generic_event_device.c  | 15 +++++++++++++++
 hw/acpi/ich9.c                  | 15 +++++++++++++++
 hw/acpi/pcihp.c                 | 17 +++++++++++++++++
 hw/acpi/piix4.c                 | 15 +++++++++++++++
 hw/core/hotplug.c               | 11 +++++++++++
 hw/isa/lpc_ich9.c               |  1 +
 hw/pci/pcie.c                   | 20 ++++++++++++++++++++
 hw/pci/pcie_port.c              |  1 +
 hw/s390x/s390-pci-bus.c         |  4 ++--
 hw/vfio/ap.c                    |  2 +-
 hw/vfio/ccw.c                   |  2 +-
 hw/vfio/pci.c                   |  2 +-
 hw/xen/xen-legacy-backend.c     |  2 +-
 hw/xen/xen_pvdev.c              |  2 +-
 include/hw/acpi/ich9.h          |  2 ++
 include/hw/acpi/pcihp.h         |  3 +++
 include/hw/core/hotplug.h       | 12 ++++++++++++
 include/hw/core/qdev.h          |  2 +-
 include/hw/pci/pcie.h           |  2 ++
 qapi/qdev.json                  | 11 +++++++++--
 system/qdev-monitor.c           | 23 +++++++++++++++++------
 23 files changed, 160 insertions(+), 21 deletions(-)

base-commit: eea8fe61b8be8f3016e522e6af24924a0266ca95

Thank you very much!

Dongli Zhang

[PATCH 1/8] qdev: Add force argument to qdev_unplug

[Dongli Zhang]

Add a force argument to qdev_unplug() and update all existing callers to
pass false.

No functional change.

The upcoming patches will add the hotplug controller callback used to
implement it to force detach a PCI device.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 hw/s390x/s390-pci-bus.c     | 4 ++--
 hw/vfio/ap.c                | 2 +-
 hw/vfio/ccw.c               | 2 +-
 hw/vfio/pci.c               | 2 +-
 hw/xen/xen-legacy-backend.c | 2 +-
 hw/xen/xen_pvdev.c          | 2 +-
 include/hw/core/qdev.h      | 2 +-
 system/qdev-monitor.c       | 4 ++--
 8 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/hw/s390x/s390-pci-bus.c b/hw/s390x/s390-pci-bus.c
index eff980fdfe..11ff9bcce5 100644
--- a/hw/s390x/s390-pci-bus.c
+++ b/hw/s390x/s390-pci-bus.c
@@ -1276,7 +1276,7 @@ static void s390_pcihost_unplug_request(HotplugHandler *hotplug_dev,
         }
 
         pbdev->pci_unplug_request_processed = true;
-        qdev_unplug(DEVICE(pbdev), errp);
+        qdev_unplug(DEVICE(pbdev), false, errp);
     } else if (object_dynamic_cast(OBJECT(dev), TYPE_S390_PCI_DEVICE)) {
         pbdev = S390_PCI_DEVICE(dev);
 
@@ -1287,7 +1287,7 @@ static void s390_pcihost_unplug_request(HotplugHandler *hotplug_dev,
          * is not blocked, e.g. because it's a PCI bridge).
          */
         if (pbdev->pdev && !pbdev->pci_unplug_request_processed) {
-            qdev_unplug(DEVICE(pbdev->pdev), errp);
+            qdev_unplug(DEVICE(pbdev->pdev), false, errp);
             return;
         }
         pbdev->pci_unplug_request_processed = false;
diff --git a/hw/vfio/ap.c b/hw/vfio/ap.c
index 6e2a1223ea..8e7c72dc8b 100644
--- a/hw/vfio/ap.c
+++ b/hw/vfio/ap.c
@@ -79,7 +79,7 @@ static void vfio_ap_req_notifier_handler(void *opaque)
         return;
     }
 
-    qdev_unplug(DEVICE(vapdev), &err);
+    qdev_unplug(DEVICE(vapdev), false, &err);
 
     if (err) {
         warn_reportf_err(err, VFIO_MSG_PREFIX, vapdev->vdev.name);
diff --git a/hw/vfio/ccw.c b/hw/vfio/ccw.c
index c3dc7c1962..c7d48966dc 100644
--- a/hw/vfio/ccw.c
+++ b/hw/vfio/ccw.c
@@ -282,7 +282,7 @@ static void vfio_ccw_req_notifier_handler(void *opaque)
         return;
     }
 
-    qdev_unplug(DEVICE(vcdev), &err);
+    qdev_unplug(DEVICE(vcdev), false, &err);
     if (err) {
         warn_reportf_err(err, VFIO_MSG_PREFIX, vcdev->vdev.name);
     }
diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
index 428ab2f069..aafa841241 100644
--- a/hw/vfio/pci.c
+++ b/hw/vfio/pci.c
@@ -3328,7 +3328,7 @@ static void vfio_req_notifier_handler(void *opaque)
         return;
     }
 
-    qdev_unplug(DEVICE(vdev), &err);
+    qdev_unplug(DEVICE(vdev), false, &err);
     if (err) {
         warn_reportf_err(err, VFIO_MSG_PREFIX, vdev->vbasedev.name);
     }
diff --git a/hw/xen/xen-legacy-backend.c b/hw/xen/xen-legacy-backend.c
index 7977b52712..4aa0339887 100644
--- a/hw/xen/xen-legacy-backend.c
+++ b/hw/xen/xen-legacy-backend.c
@@ -186,7 +186,7 @@ static struct XenLegacyDevice *xen_be_get_xendev(const char *type, int dom,
     xendev->evtchndev = qemu_xen_evtchn_open();
     if (xendev->evtchndev == NULL) {
         xen_pv_printf(NULL, 0, "can't open evtchn device\n");
-        qdev_unplug(DEVICE(xendev), NULL);
+        qdev_unplug(DEVICE(xendev), false, NULL);
         return NULL;
     }
     qemu_set_cloexec(qemu_xen_evtchn_fd(xendev->evtchndev));
diff --git a/hw/xen/xen_pvdev.c b/hw/xen/xen_pvdev.c
index e36370e2ee..9518f5b3b5 100644
--- a/hw/xen/xen_pvdev.c
+++ b/hw/xen/xen_pvdev.c
@@ -273,7 +273,7 @@ void xen_pv_del_xendev(struct XenLegacyDevice *xendev)
 
     QTAILQ_REMOVE(&xendevs, xendev, next);
 
-    qdev_unplug(DEVICE(xendev), NULL);
+    qdev_unplug(DEVICE(xendev), false, NULL);
 }
 
 void xen_pv_insert_xendev(struct XenLegacyDevice *xendev)
diff --git a/include/hw/core/qdev.h b/include/hw/core/qdev.h
index 37f7d33551..c1daa74914 100644
--- a/include/hw/core/qdev.h
+++ b/include/hw/core/qdev.h
@@ -527,7 +527,7 @@ bool qdev_hotunplug_allowed(DeviceState *dev, Error **errp);
  * or NULL if there aren't any.
  */
 HotplugHandler *qdev_get_hotplug_handler(DeviceState *dev);
-void qdev_unplug(DeviceState *dev, Error **errp);
+void qdev_unplug(DeviceState *dev, bool force, Error **errp);
 int qdev_sync_config(DeviceState *dev, Error **errp);
 void qdev_simple_device_unplug_cb(HotplugHandler *hotplug_dev,
                                   DeviceState *dev, Error **errp);
diff --git a/system/qdev-monitor.c b/system/qdev-monitor.c
index 00fed791cc..3606a347a0 100644
--- a/system/qdev-monitor.c
+++ b/system/qdev-monitor.c
@@ -912,7 +912,7 @@ static DeviceState *find_device_state(const char *id, bool use_generic_error,
     return dev;
 }
 
-void qdev_unplug(DeviceState *dev, Error **errp)
+void qdev_unplug(DeviceState *dev, bool force, Error **errp)
 {
     HotplugHandler *hotplug_ctrl;
     HotplugHandlerClass *hdc;
@@ -960,7 +960,7 @@ void qmp_device_del(const char *id, Error **errp)
             return;
         }
 
-        qdev_unplug(dev, errp);
+        qdev_unplug(dev, false, errp);
     }
 }
 
-- 
2.43.5

[PATCH 2/8] qdev: hotplug: Add force_unplug handler callback

[Dongli Zhang]

Add a HotplugHandlerClass force_unplug callback and a wrapper used by the
generic qdev unplug path.

When qdev_unplug() is called with force=true, delegate to the hotplug
controller if it implements force_unplug. Controllers without the callback
return a normal error. This makes forced removal available only for
controllers that explicitly implement the operation.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 hw/core/hotplug.c         | 11 +++++++++++
 include/hw/core/hotplug.h | 12 ++++++++++++
 system/qdev-monitor.c     | 10 +++++++++-
 3 files changed, 32 insertions(+), 1 deletion(-)

diff --git a/hw/core/hotplug.c b/hw/core/hotplug.c
index 68aabad8ae..0e44d98cf7 100644
--- a/hw/core/hotplug.c
+++ b/hw/core/hotplug.c
@@ -57,6 +57,17 @@ void hotplug_handler_unplug(HotplugHandler *plug_handler,
     }
 }
 
+void hotplug_handler_force_unplug(HotplugHandler *plug_handler,
+                                  DeviceState *plugged_dev,
+                                  Error **errp)
+{
+    HotplugHandlerClass *hdc = HOTPLUG_HANDLER_GET_CLASS(plug_handler);
+
+    if (hdc->force_unplug) {
+        hdc->force_unplug(plug_handler, plugged_dev, errp);
+    }
+}
+
 static const TypeInfo hotplug_handler_info = {
     .name          = TYPE_HOTPLUG_HANDLER,
     .parent        = TYPE_INTERFACE,
diff --git a/include/hw/core/hotplug.h b/include/hw/core/hotplug.h
index a9840ed485..300ac4fa8f 100644
--- a/include/hw/core/hotplug.h
+++ b/include/hw/core/hotplug.h
@@ -48,6 +48,8 @@ typedef void (*hotplug_fn)(HotplugHandler *plug_handler,
  * @unplug: unplug callback.
  *          Used for device removal with devices that implement
  *          asynchronous and synchronous (surprise) removal.
+ * @force_unplug: force unplug callback.
+ *                Used to complete enforced removal without guest cooperation.
  * @is_hotpluggable_bus: called to check if bus/its parent allow hotplug on bus
  */
 struct HotplugHandlerClass {
@@ -59,6 +61,7 @@ struct HotplugHandlerClass {
     hotplug_fn plug;
     hotplug_fn unplug_request;
     hotplug_fn unplug;
+    hotplug_fn force_unplug;
     bool (*is_hotpluggable_bus)(HotplugHandler *plug_handler, BusState *bus);
 };
 
@@ -96,4 +99,13 @@ void hotplug_handler_unplug_request(HotplugHandler *plug_handler,
 void hotplug_handler_unplug(HotplugHandler *plug_handler,
                             DeviceState *plugged_dev,
                             Error **errp);
+
+/**
+ * hotplug_handler_force_unplug:
+ *
+ * Calls #HotplugHandlerClass.force_unplug callback of @plug_handler.
+ */
+void hotplug_handler_force_unplug(HotplugHandler *plug_handler,
+                                  DeviceState *plugged_dev,
+                                  Error **errp);
 #endif
diff --git a/system/qdev-monitor.c b/system/qdev-monitor.c
index 3606a347a0..fa3cae246b 100644
--- a/system/qdev-monitor.c
+++ b/system/qdev-monitor.c
@@ -937,7 +937,15 @@ void qdev_unplug(DeviceState *dev, bool force, Error **errp)
     /* If device supports async unplug just request it to be done,
      * otherwise just remove it synchronously */
     hdc = HOTPLUG_HANDLER_GET_CLASS(hotplug_ctrl);
-    if (hdc->unplug_request) {
+
+    if (force) {
+        if (!hdc->force_unplug) {
+            error_setg(&local_err, "Device '%s' does not support forced unplug",
+                       dev->id ? dev->id : object_get_typename(OBJECT(dev)));
+        } else {
+            hotplug_handler_force_unplug(hotplug_ctrl, dev, &local_err);
+        }
+    } else if (hdc->unplug_request) {
         hotplug_handler_unplug_request(hotplug_ctrl, dev, &local_err);
     } else {
         hotplug_handler_unplug(hotplug_ctrl, dev, &local_err);
-- 
2.43.5

[PATCH 3/8] qdev: Support forced device_del in QMP and HMP

[Dongli Zhang]

Add an optional force argument to the QMP device_del command and expose it
in HMP as "device_del -f".

When force is requested, qdev_unplug() bypasses the pending deletion guard
and asks the selected hotplug controller to complete removal through its
force_unplug callback. Controllers that do not implement the callback
reject the operation.

Forced removal bypasses guest cooperation.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 hmp-commands.hx       | 11 ++++++-----
 qapi/qdev.json        | 11 +++++++++--
 system/qdev-monitor.c | 11 +++++++----
 3 files changed, 22 insertions(+), 11 deletions(-)

diff --git a/hmp-commands.hx b/hmp-commands.hx
index 43ff220b5f..022502b20f 100644
--- a/hmp-commands.hx
+++ b/hmp-commands.hx
@@ -708,17 +708,18 @@ ERST
 
     {
         .name       = "device_del",
-        .args_type  = "id:s",
-        .params     = "device",
-        .help       = "remove device",
+        .args_type  = "force:-f,id:s",
+        .params     = "[-f] device",
+        .help       = "remove device, use -f to force removal",
         .cmd        = hmp_device_del,
         .command_completion = device_del_completion,
     },
 
 SRST
-``device_del`` *id*
+``device_del`` [*-f*] *id*
   Remove device *id*. *id* may be a short ID
-  or a QOM object path.
+  or a QOM object path.  Use -f to force removal without waiting for
+  guest cooperation.
 ERST
 
     {
diff --git a/qapi/qdev.json b/qapi/qdev.json
index 974cf9c583..cb5b5ad1db 100644
--- a/qapi/qdev.json
+++ b/qapi/qdev.json
@@ -90,6 +90,11 @@
 #
 # @id: the device's ID or QOM path
 #
+# @force: if true, remove the device without waiting for guest
+#     cooperation.  The guest may still be using the device.  This can
+#     cause guest-visible errors, I/O failures, or guest crashes.
+#     (since 11.2)
+#
 # Errors:
 #     - If @id is not a valid device, DeviceNotFound
 #
@@ -101,7 +106,9 @@
 #    will automatically complete removal for all devices.  If a
 #    guest-side error in the hot removal process is detected, the
 #    device will not be removed and a `DEVICE_UNPLUG_GUEST_ERROR`
-#    event is sent.  Some errors cannot be detected.
+#    event is sent.  Some errors cannot be detected.  If @force is
+#    true, guest cooperation is bypassed, but backend cleanup is still
+#    performed through the device's normal unrealize path.
 #
 # Since: 0.14
 #
@@ -117,7 +124,7 @@
 #          "arguments": { "id": "/machine/peripheral-anon/device[0]" } }
 #     <- { "return": {} }
 ##
-{ 'command': 'device_del', 'data': {'id': 'str'} }
+{ 'command': 'device_del', 'data': {'id': 'str', '*force': 'bool'} }
 
 ##
 # @DEVICE_DELETED:
diff --git a/system/qdev-monitor.c b/system/qdev-monitor.c
index fa3cae246b..ca10a25c46 100644
--- a/system/qdev-monitor.c
+++ b/system/qdev-monitor.c
@@ -956,11 +956,13 @@ void qdev_unplug(DeviceState *dev, bool force, Error **errp)
     error_propagate(errp, local_err);
 }
 
-void qmp_device_del(const char *id, Error **errp)
+void qmp_device_del(const char *id, bool has_force, bool force, Error **errp)
 {
     DeviceState *dev = find_device_state(id, false, errp);
+    bool do_force = has_force && force;
+
     if (dev != NULL) {
-        if (dev->pending_deleted_event &&
+        if (!do_force && dev->pending_deleted_event &&
             (dev->pending_deleted_expires_ms == 0 ||
              dev->pending_deleted_expires_ms > qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL))) {
             error_setg(errp, "Device %s is already in the "
@@ -968,7 +970,7 @@ void qmp_device_del(const char *id, Error **errp)
             return;
         }
 
-        qdev_unplug(dev, false, errp);
+        qdev_unplug(dev, do_force, errp);
     }
 }
 
@@ -1046,9 +1048,10 @@ out:
 void hmp_device_del(Monitor *mon, const QDict *qdict)
 {
     const char *id = qdict_get_str(qdict, "id");
+    bool force = qdict_get_try_bool(qdict, "force", false);
     Error *err = NULL;
 
-    qmp_device_del(id, &err);
+    qmp_device_del(id, true, force, &err);
     hmp_handle_error(mon, err);
 }
 
-- 
2.43.5

[PATCH 4/8] hw/acpi/pcihp: Add forced slot unplug helper

[Dongli Zhang]

Add acpi_pcihp_device_force_unplug_cb() to complete PCI slot removal
through the existing ACPI PCI hotplug eject path.

Forced unplug intentionally does not introduce a separate teardown
sequence. It lets management trigger the same slot completion helper that
is already reachable when the guest writes the ACPI EJ register.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 hw/acpi/acpi-pci-hotplug-stub.c |  6 ++++++
 hw/acpi/pcihp.c                 | 17 +++++++++++++++++
 include/hw/acpi/pcihp.h         |  3 +++
 3 files changed, 26 insertions(+)

diff --git a/hw/acpi/acpi-pci-hotplug-stub.c b/hw/acpi/acpi-pci-hotplug-stub.c
index d58ea726a8..9451a45c0b 100644
--- a/hw/acpi/acpi-pci-hotplug-stub.c
+++ b/hw/acpi/acpi-pci-hotplug-stub.c
@@ -30,6 +30,12 @@ void acpi_pcihp_device_unplug_request_cb(HotplugHandler *hotplug_dev,
 {
 }
 
+void acpi_pcihp_device_force_unplug_cb(HotplugHandler *hotplug_dev,
+                                       AcpiPciHpState *s, DeviceState *dev,
+                                       Error **errp)
+{
+}
+
 void acpi_pcihp_reset(AcpiPciHpState *s)
 {
 }
diff --git a/hw/acpi/pcihp.c b/hw/acpi/pcihp.c
index 87162ff2c0..7a1f2b873a 100644
--- a/hw/acpi/pcihp.c
+++ b/hw/acpi/pcihp.c
@@ -367,6 +367,23 @@ void acpi_pcihp_device_unplug_request_cb(HotplugHandler *hotplug_dev,
     acpi_send_event(DEVICE(hotplug_dev), ACPI_PCI_HOTPLUG_STATUS);
 }
 
+void acpi_pcihp_device_force_unplug_cb(HotplugHandler *hotplug_dev,
+                                       AcpiPciHpState *s, DeviceState *dev,
+                                       Error **errp)
+{
+    PCIDevice *pdev = PCI_DEVICE(dev);
+    int slot = PCI_SLOT(pdev->devfn);
+    int bsel = acpi_pcihp_get_bsel(pci_get_bus(pdev));
+
+    if (bsel < 0) {
+        error_setg(errp, "Unsupported bus. Bus doesn't have property '"
+                   ACPI_PCIHP_PROP_BSEL "' set");
+        return;
+    }
+
+    acpi_pcihp_eject_slot(s, bsel, 1U << slot);
+}
+
 bool acpi_pcihp_is_hotpluggable_bus(AcpiPciHpState *s, BusState *bus)
 {
     Object *o = OBJECT(bus->parent);
diff --git a/include/hw/acpi/pcihp.h b/include/hw/acpi/pcihp.h
index efce5fd2e1..13c131b8e8 100644
--- a/include/hw/acpi/pcihp.h
+++ b/include/hw/acpi/pcihp.h
@@ -75,6 +75,9 @@ void acpi_pcihp_device_unplug_cb(HotplugHandler *hotplug_dev, AcpiPciHpState *s,
 void acpi_pcihp_device_unplug_request_cb(HotplugHandler *hotplug_dev,
                                          AcpiPciHpState *s, DeviceState *dev,
                                          Error **errp);
+void acpi_pcihp_device_force_unplug_cb(HotplugHandler *hotplug_dev,
+                                       AcpiPciHpState *s, DeviceState *dev,
+                                       Error **errp);
 
 void build_acpi_pci_hotplug(Aml *table, AmlRegionSpace rs, uint64_t pcihp_addr);
 void build_append_pci_dsm_func0_common(Aml *ctx, Aml *retvar);
-- 
2.43.5

[PATCH 5/8] hw/acpi/piix4: Support forced PCI unplug

[Dongli Zhang]

Support forced PCI unplug for PIIX4. This enables forced removal for ACPI
PCI hotplug on i440fx machines while rejecting non-PCI devices.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 hw/acpi/piix4.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/hw/acpi/piix4.c b/hw/acpi/piix4.c
index 9b7f50c7af..7cfdcb7935 100644
--- a/hw/acpi/piix4.c
+++ b/hw/acpi/piix4.c
@@ -383,6 +383,20 @@ static void piix4_device_unplug_cb(HotplugHandler *hotplug_dev,
     }
 }
 
+static void piix4_device_force_unplug_cb(HotplugHandler *hotplug_dev,
+                                         DeviceState *dev, Error **errp)
+{
+    PIIX4PMState *s = PIIX4_PM(hotplug_dev);
+
+    if (object_dynamic_cast(OBJECT(dev), TYPE_PCI_DEVICE)) {
+        acpi_pcihp_device_force_unplug_cb(hotplug_dev, &s->acpi_pci_hotplug,
+                                          dev, errp);
+    } else {
+        error_setg(errp, "acpi: forced device unplug for not supported device"
+                   " type: %s", object_get_typename(OBJECT(dev)));
+    }
+}
+
 static bool piix4_is_hotpluggable_bus(HotplugHandler *hotplug_dev,
                                       BusState *bus)
 {
@@ -604,6 +618,7 @@ static void piix4_pm_class_init(ObjectClass *klass, const void *data)
     hc->plug = piix4_device_plug_cb;
     hc->unplug_request = piix4_device_unplug_request_cb;
     hc->unplug = piix4_device_unplug_cb;
+    hc->force_unplug = piix4_device_force_unplug_cb;
     hc->is_hotpluggable_bus = piix4_is_hotpluggable_bus;
     adevc->ospm_status = piix4_ospm_status;
     adevc->send_event = piix4_send_gpe;
-- 
2.43.5

[PATCH 6/8] hw/acpi/ich9: Support forced PCI unplug

[Dongli Zhang]

Support forced PCI unplug for ICH9. This enables forced removal for ACPI
PCI hotplug on q35 machines while rejecting non-PCI devices.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 hw/acpi/ich9.c         | 15 +++++++++++++++
 hw/isa/lpc_ich9.c      |  1 +
 include/hw/acpi/ich9.h |  2 ++
 3 files changed, 18 insertions(+)

diff --git a/hw/acpi/ich9.c b/hw/acpi/ich9.c
index 5e8f8a7eaf..4bcaf3858b 100644
--- a/hw/acpi/ich9.c
+++ b/hw/acpi/ich9.c
@@ -528,6 +528,21 @@ void ich9_pm_device_unplug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
     }
 }
 
+void ich9_pm_device_force_unplug_cb(HotplugHandler *hotplug_dev,
+                                    DeviceState *dev, Error **errp)
+{
+    ICH9LPCState *lpc = ICH9_LPC_DEVICE(hotplug_dev);
+
+    if (object_dynamic_cast(OBJECT(dev), TYPE_PCI_DEVICE)) {
+        acpi_pcihp_device_force_unplug_cb(hotplug_dev,
+                                          &lpc->pm.acpi_pci_hotplug,
+                                          dev, errp);
+    } else {
+        error_setg(errp, "acpi: forced device unplug for not supported device"
+                   " type: %s", object_get_typename(OBJECT(dev)));
+    }
+}
+
 bool ich9_pm_is_hotpluggable_bus(HotplugHandler *hotplug_dev, BusState *bus)
 {
     ICH9LPCState *lpc = ICH9_LPC_DEVICE(hotplug_dev);
diff --git a/hw/isa/lpc_ich9.c b/hw/isa/lpc_ich9.c
index edf9783ec8..6d734ffc47 100644
--- a/hw/isa/lpc_ich9.c
+++ b/hw/isa/lpc_ich9.c
@@ -908,6 +908,7 @@ static void ich9_lpc_class_init(ObjectClass *klass, const void *data)
     hc->plug = ich9_pm_device_plug_cb;
     hc->unplug_request = ich9_pm_device_unplug_request_cb;
     hc->unplug = ich9_pm_device_unplug_cb;
+    hc->force_unplug = ich9_pm_device_force_unplug_cb;
     hc->is_hotpluggable_bus = ich9_pm_is_hotpluggable_bus;
     adevc->ospm_status = ich9_pm_ospm_status;
     adevc->send_event = ich9_send_gpe;
diff --git a/include/hw/acpi/ich9.h b/include/hw/acpi/ich9.h
index 30990fcef5..cc9f50a60f 100644
--- a/include/hw/acpi/ich9.h
+++ b/include/hw/acpi/ich9.h
@@ -93,6 +93,8 @@ void ich9_pm_device_unplug_request_cb(HotplugHandler *hotplug_dev,
                                       DeviceState *dev, Error **errp);
 void ich9_pm_device_unplug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
                               Error **errp);
+void ich9_pm_device_force_unplug_cb(HotplugHandler *hotplug_dev,
+                                    DeviceState *dev, Error **errp);
 bool ich9_pm_is_hotpluggable_bus(HotplugHandler *hotplug_dev, BusState *bus);
 
 void ich9_pm_ospm_status(AcpiDeviceIf *adev, ACPIOSTInfoList ***list);
-- 
2.43.5

[PATCH 7/8] hw/acpi/ged: Support forced PCI unplug

[Dongli Zhang]

Support forced PCI unplug for generic event device (ged). This enables
forced removal for ACPI PCI hotplug on arm64 virt machines while rejecting
non-PCI devices.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 hw/acpi/generic_event_device.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/hw/acpi/generic_event_device.c b/hw/acpi/generic_event_device.c
index 9e9416d406..3c041f5c26 100644
--- a/hw/acpi/generic_event_device.c
+++ b/hw/acpi/generic_event_device.c
@@ -314,6 +314,20 @@ static void acpi_ged_unplug_cb(HotplugHandler *hotplug_dev,
     }
 }
 
+static void acpi_ged_force_unplug_cb(HotplugHandler *hotplug_dev,
+                                     DeviceState *dev, Error **errp)
+{
+    AcpiGedState *s = ACPI_GED(hotplug_dev);
+
+    if (object_dynamic_cast(OBJECT(dev), TYPE_PCI_DEVICE)) {
+        acpi_pcihp_device_force_unplug_cb(hotplug_dev, &s->pcihp_state,
+                                          dev, errp);
+    } else {
+        error_setg(errp, "acpi: forced device unplug for unsupported device"
+                   " type: %s", object_get_typename(OBJECT(dev)));
+    }
+}
+
 static void acpi_ged_ospm_status(AcpiDeviceIf *adev, ACPIOSTInfoList ***list)
 {
     AcpiGedState *s = ACPI_GED(adev);
@@ -603,6 +617,7 @@ static void acpi_ged_class_init(ObjectClass *class, const void *data)
     hc->plug = acpi_ged_device_plug_cb;
     hc->unplug_request = acpi_ged_unplug_request_cb;
     hc->unplug = acpi_ged_unplug_cb;
+    hc->force_unplug = acpi_ged_force_unplug_cb;
     resettable_class_set_parent_phases(rc, NULL, ged_reset_hold, NULL,
                                        &gedc->parent_phases);
 
-- 
2.43.5

[PATCH 8/8] hw/pci/pcie: Support forced PCIe native unplug

[Dongli Zhang]

Support forced PCIe native unplug by implementing force_unplug method.

The forced path completes the pending slot unplug immediately through the
existing PCIe slot removal helper, clears the attention button pressed
status bit, and notifies the hotplug event.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 hw/pci/pcie.c         | 20 ++++++++++++++++++++
 hw/pci/pcie_port.c    |  1 +
 include/hw/pci/pcie.h |  2 ++
 3 files changed, 23 insertions(+)

diff --git a/hw/pci/pcie.c b/hw/pci/pcie.c
index 4622c75e48..bd967ca73b 100644
--- a/hw/pci/pcie.c
+++ b/hw/pci/pcie.c
@@ -666,6 +666,26 @@ void pcie_cap_slot_unplug_request_cb(HotplugHandler *hotplug_dev,
     pcie_cap_slot_push_attention_button(hotplug_pdev);
 }
 
+void pcie_cap_slot_force_unplug_cb(HotplugHandler *hotplug_dev,
+                                   DeviceState *dev, Error **errp)
+{
+    PCIDevice *hotplug_pdev = PCI_DEVICE(hotplug_dev);
+    uint8_t *exp_cap = hotplug_pdev->config + hotplug_pdev->exp.exp_cap;
+    uint32_t sltcap = pci_get_word(exp_cap + PCI_EXP_SLTCAP);
+
+    if ((sltcap & PCI_EXP_SLTCAP_HPC) == 0) {
+        error_setg(errp, "Hot-unplug failed: "
+                   "unsupported by the port device '%s'",
+                   DEVICE(hotplug_pdev)->id);
+        return;
+    }
+
+    pcie_cap_slot_do_unplug(hotplug_pdev);
+    pci_word_test_and_clear_mask(exp_cap + PCI_EXP_SLTSTA,
+                                 PCI_EXP_SLTSTA_ABP);
+    hotplug_event_notify(hotplug_pdev);
+}
+
 /* pci express slot for pci express root/downstream port
    PCI express capability slot registers */
 void pcie_cap_slot_init(PCIDevice *dev, PCIESlot *s)
diff --git a/hw/pci/pcie_port.c b/hw/pci/pcie_port.c
index dbb6032160..4806c8289b 100644
--- a/hw/pci/pcie_port.c
+++ b/hw/pci/pcie_port.c
@@ -221,6 +221,7 @@ static void pcie_slot_class_init(ObjectClass *oc, const void *data)
     hc->plug = pcie_cap_slot_plug_cb;
     hc->unplug = pcie_cap_slot_unplug_cb;
     hc->unplug_request = pcie_cap_slot_unplug_request_cb;
+    hc->force_unplug = pcie_cap_slot_force_unplug_cb;
     hc->is_hotpluggable_bus = pcie_slot_is_hotpluggable_bus;
 }
 
diff --git a/include/hw/pci/pcie.h b/include/hw/pci/pcie.h
index 71ba94874b..08c6c81f34 100644
--- a/include/hw/pci/pcie.h
+++ b/include/hw/pci/pcie.h
@@ -154,6 +154,8 @@ void pcie_cap_slot_unplug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
                              Error **errp);
 void pcie_cap_slot_unplug_request_cb(HotplugHandler *hotplug_dev,
                                      DeviceState *dev, Error **errp);
+void pcie_cap_slot_force_unplug_cb(HotplugHandler *hotplug_dev,
+                                   DeviceState *dev, Error **errp);
 
 void pcie_pasid_common_init(PCIDevice *dev, uint16_t offset,
                             uint8_t pasid_width, bool exec_perm, bool priv_mod);
-- 
2.43.5

Re: [PATCH 1/8] qdev: Add force argument to qdev_unplug

[Jason J. Herne]

On 8/23/26 9:13 PM, Dongli Zhang wrote:
> ...
> diff --git a/hw/vfio/ap.c b/hw/vfio/ap.c
> index 6e2a1223ea..8e7c72dc8b 100644
> --- a/hw/vfio/ap.c
> +++ b/hw/vfio/ap.c
> @@ -79,7 +79,7 @@ static void vfio_ap_req_notifier_handler(void *opaque)
>           return;
>       }
>   
> -    qdev_unplug(DEVICE(vapdev), &err);
> +    qdev_unplug(DEVICE(vapdev), false, &err);
>   
>       if (err) {
>           warn_reportf_err(err, VFIO_MSG_PREFIX, vapdev->vdev.name);
Reviewed-by: Jason J. Herne <jjherne@linux.ibm.com>

Re: [PATCH 3/8] qdev: Support forced device_del in QMP and HMP

[Daniel P. Berrangé]

On Sun, Aug 23, 2026 at 06:13:33PM -0700, Dongli Zhang wrote:
> Add an optional force argument to the QMP device_del command and expose it
> in HMP as "device_del -f".
> 
> When force is requested, qdev_unplug() bypasses the pending deletion guard
> and asks the selected hotplug controller to complete removal through its
> force_unplug callback. Controllers that do not implement the callback
> reject the operation.
> 
> Forced removal bypasses guest cooperation.

This sentence is rather missing the punchline....

  Force removal bypasses guest cooperation and may result in guest
  errors, I/O failures, or guest panics. The guest OS cannot be
  trusted after force removal until a full power cycle has been
  performed.

I'm rather on the fence as to whether it is a good idea to enable
this feature or not. If it is used by a cloud admin without knowledge
of the guest owner, its use is liable to lead to hard-to-debug/diagnose
problems in the guest OS.

If a guest OS is not honouring an unplug request and the host owner needs
to force reclaim a resource, power off is always there as the failsafe.

> diff --git a/qapi/qdev.json b/qapi/qdev.json
> index 974cf9c583..cb5b5ad1db 100644
> --- a/qapi/qdev.json
> +++ b/qapi/qdev.json
> @@ -90,6 +90,11 @@
>  #
>  # @id: the device's ID or QOM path
>  #
> +# @force: if true, remove the device without waiting for guest
> +#     cooperation.  The guest may still be using the device.  This can
> +#     cause guest-visible errors, I/O failures, or guest crashes.

I'd want to be warning in a stronger way.


 This is a dangerous operation that can cause guest-visible errors,
 I/O failures, or guest crashes. The guest OS state should not be
 trusted after a forced device removal, until a full power cycle has
 been performed.


> +#     (since 11.2)
> +#
>  # Errors:
>  #     - If @id is not a valid device, DeviceNotFound
>  #
> @@ -101,7 +106,9 @@
>  #    will automatically complete removal for all devices.  If a
>  #    guest-side error in the hot removal process is detected, the
>  #    device will not be removed and a `DEVICE_UNPLUG_GUEST_ERROR`
> -#    event is sent.  Some errors cannot be detected.
> +#    event is sent.  Some errors cannot be detected.  If @force is
> +#    true, guest cooperation is bypassed, but backend cleanup is still
> +#    performed through the device's normal unrealize path.
>  #
>  # Since: 0.14
>  #
> @@ -117,7 +124,7 @@
>  #          "arguments": { "id": "/machine/peripheral-anon/device[0]" } }
>  #     <- { "return": {} }
>  ##
> -{ 'command': 'device_del', 'data': {'id': 'str'} }
> +{ 'command': 'device_del', 'data': {'id': 'str', '*force': 'bool'} }
>  
>  ##
>  # @DEVICE_DELETED:
> diff --git a/system/qdev-monitor.c b/system/qdev-monitor.c
> index fa3cae246b..ca10a25c46 100644
> --- a/system/qdev-monitor.c
> +++ b/system/qdev-monitor.c
> @@ -956,11 +956,13 @@ void qdev_unplug(DeviceState *dev, bool force, Error **errp)
>      error_propagate(errp, local_err);
>  }

With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Re: [PATCH 0/8] Force detach PCI devices for ACPI-based and PCIe native hot-unplug

[Daniel P. Berrangé]

On Sun, Aug 23, 2026 at 06:13:30PM -0700, Dongli Zhang wrote:
> Hot-unplugging a PCI device can require cooperation from the guest. For
> ACPI PCI hotplug, QEMU notifies the guest through ACPI and the guest
> eventually writes the ACPI PCI eject register. For PCIe native hotplug,
> QEMU notifies the guest through the PCIe hotplug mechanism and waits for
> the slot unplug flow to complete. Only after that completion does QEMU
> unrealize the device and emit DEVICE_DELETED.
> 
> This can leave a device stuck in the unplug pending state when the guest
> does not cooperate. Examples include:
> 
> 1. The guest has panicked, or the relevant ACPI/PCI hotplug driver is
> unavailable.
> 
> 2. The guest is stalled and cannot handle the hot-unplug event. For
> example, stalling the Linux [irq/9-acpi] kernel thread can reproduce this
> for ACPI-based hot-unplug.
> 
> 3. The device was attached to a slot that the guest cannot use. For
> example, a pcie-root-port only supports slot 0. If a device is added to a
> non-zero slot below a pcie-root-port, the guest may never discover the
> device and therefore may never complete the unplug request.
> 
> The non-zero slot case has also been discussed in:
> 
> hw/pci: warn when PCIe device is plugged into non-zero slot of downstream port
> https://gitlab.com/qemu-project/qemu/-/commit/ca92eb5defcf9d1c2106341744a73a03cf26e824
> 
> hw/pci: add comment to explain checking for available function 0 in pci hotplug
> https://gitlab.com/qemu-project/qemu/-/commit/67d045a0ef5b9c5f871c3a1d87325a8a42d2b9d5
> 
> pci: don't skip function 0 occupancy verification for devfn auto assign
> https://gitlab.com/qemu-project/qemu/-/commit/e228d62b4af29bca698ec57efdceb46f392f5444
> 
> For example, if root-port.1 is a pcie-root-port, the following command adds
> a vhost-scsi-pci device to an invalid slot:
> 
> (qemu) device_add vhost-scsi-pci,id=scsi01,wwpn=naa.5001405324af0985,bus=root-port.1,addr=01.0
> warning: PCI: slot 1 is not valid for vhost-scsi-pci, parent device only allows plugging into slot 0.

This rather looks like it should be a fatal error, not a mere warning.

If I follow the commit ca92eb5def it links to https://bugzilla.redhat.com/show_bug.cgi?id=2128929
which states that this configuration is going to lead to a crash in
QEMU on guest OS shutdown. IMHO that crash is sufficient to justify
making this a fatal error.

If we actually wanted this to remain a warning, then that shutdown
crash would need to be fixed.

> 
> In this situation, the device may be impossible to remove through normal
> guest-cooperative hot-unplug. Production environments also need a host side
> recovery option when the guest kernel is the reason that unplug does not
> complete.
> 
> This series adds a force option to QMP device_del and HMP device_del. When
> requested, QEMU asks the selected hotplug controller to complete the unplug
> through a new force_unplug callback.
> 
> This series implements forced unplug for ACPI PCI hotplug and PCIe native
> hotplug, which cover common pc/q35/virt cases. SHPC is not implemented by
> this series.
> 
> 
> Dongli Zhang (8):
>   qdev: Add force argument to qdev_unplug
>   qdev: hotplug: Add force_unplug handler callback
>   qdev: Support forced device_del in QMP and HMP
>   hw/acpi/pcihp: acpi/pcihp: Add forced slot unplug helper
>   hw/acpi/piix4: Support forced PCI unplug
>   hw/acpi/ich9: Support forced PCI unplug
>   hw/acpi/ged: Support forced PCI unplug
>   hw/pci/pcie: Support forced PCIe native unplug
> 
>  hmp-commands.hx                 | 11 ++++++-----
>  hw/acpi/acpi-pci-hotplug-stub.c |  6 ++++++
>  hw/acpi/generic_event_device.c  | 15 +++++++++++++++
>  hw/acpi/ich9.c                  | 15 +++++++++++++++
>  hw/acpi/pcihp.c                 | 17 +++++++++++++++++
>  hw/acpi/piix4.c                 | 15 +++++++++++++++
>  hw/core/hotplug.c               | 11 +++++++++++
>  hw/isa/lpc_ich9.c               |  1 +
>  hw/pci/pcie.c                   | 20 ++++++++++++++++++++
>  hw/pci/pcie_port.c              |  1 +
>  hw/s390x/s390-pci-bus.c         |  4 ++--
>  hw/vfio/ap.c                    |  2 +-
>  hw/vfio/ccw.c                   |  2 +-
>  hw/vfio/pci.c                   |  2 +-
>  hw/xen/xen-legacy-backend.c     |  2 +-
>  hw/xen/xen_pvdev.c              |  2 +-
>  include/hw/acpi/ich9.h          |  2 ++
>  include/hw/acpi/pcihp.h         |  3 +++
>  include/hw/core/hotplug.h       | 12 ++++++++++++
>  include/hw/core/qdev.h          |  2 +-
>  include/hw/pci/pcie.h           |  2 ++
>  qapi/qdev.json                  | 11 +++++++++--
>  system/qdev-monitor.c           | 23 +++++++++++++++++------
>  23 files changed, 160 insertions(+), 21 deletions(-)
> 
> base-commit: eea8fe61b8be8f3016e522e6af24924a0266ca95
> 
> Thank you very much!
> 
> Dongli Zhang
> 

With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Re: [PATCH 3/8] qdev: Support forced device_del in QMP and HMP

[Dongli Zhang]

On Mon, Aug 24, 2026 7:42:20AM -0700, Daniel P. Berrangé wrote:
> On Sun, Aug 23, 2026 at 06:13:33PM -0700, Dongli Zhang wrote:
>> Add an optional force argument to the QMP device_del command and expose it
>> in HMP as "device_del -f".
>> 
>> When force is requested, qdev_unplug() bypasses the pending deletion guard
>> and asks the selected hotplug controller to complete removal through its
>> force_unplug callback. Controllers that do not implement the callback
>> reject the operation.
>> 
>> Forced removal bypasses guest cooperation.
> 
> This sentence is rather missing the punchline....
> 
>   Force removal bypasses guest cooperation and may result in guest
>   errors, I/O failures, or guest panics. The guest OS cannot be
>   trusted after force removal until a full power cycle has been
>   performed.
> 
> I'm rather on the fence as to whether it is a good idea to enable
> this feature or not. If it is used by a cloud admin without knowledge
> of the guest owner, its use is liable to lead to hard-to-debug/diagnose
> problems in the guest OS.
> 
> If a guest OS is not honouring an unplug request and the host owner needs
> to force reclaim a resource, power off is always there as the failsafe.

Developers with knowledge of PCI/PCIe/ACPI and system kernels (Linux, Windows,
and BSD) know that the guest VM needs to follow the appropriate protocol to
power off or eject the device from the VM side so that QEMU can safely remove
the device and send the QMP DEVICE_DELETED event to the cloud administrator.

Assume two scenarios:

1. Due to an issue with the guest VM or QEMU, the device is never detected or
used by the guest VM. Especially for ACPI-based hotplug, allowing a forced
detach enables the user or administrator to avoid having an un-detected device
stuck in the VM.

This is safe as device is never used by guest VM kernel.

2. The VM owner may expect QEMU to unplug the device even when the guest VM is
not cooperating. This provides an option with a clear warning: yes, we can
forcibly detach the PCI device, but doing so carries risks.

Thank you very much!

Dongli Zhang

> 
>> diff --git a/qapi/qdev.json b/qapi/qdev.json
>> index 974cf9c583..cb5b5ad1db 100644
>> --- a/qapi/qdev.json
>> +++ b/qapi/qdev.json
>> @@ -90,6 +90,11 @@
>>  #
>>  # @id: the device's ID or QOM path
>>  #
>> +# @force: if true, remove the device without waiting for guest
>> +#     cooperation.  The guest may still be using the device.  This can
>> +#     cause guest-visible errors, I/O failures, or guest crashes.
> 
> I'd want to be warning in a stronger way.
> 
> 
>  This is a dangerous operation that can cause guest-visible errors,
>  I/O failures, or guest crashes. The guest OS state should not be
>  trusted after a forced device removal, until a full power cycle has
>  been performed.
> 
> 
>> +#     (since 11.2)
>> +#
>>  # Errors:
>>  #     - If @id is not a valid device, DeviceNotFound
>>  #
>> @@ -101,7 +106,9 @@
>>  #    will automatically complete removal for all devices.  If a
>>  #    guest-side error in the hot removal process is detected, the
>>  #    device will not be removed and a `DEVICE_UNPLUG_GUEST_ERROR`
>> -#    event is sent.  Some errors cannot be detected.
>> +#    event is sent.  Some errors cannot be detected.  If @force is
>> +#    true, guest cooperation is bypassed, but backend cleanup is still
>> +#    performed through the device's normal unrealize path.
>>  #
>>  # Since: 0.14
>>  #
>> @@ -117,7 +124,7 @@
>>  #          "arguments": { "id": "/machine/peripheral-anon/device[0]" } }
>>  #     <- { "return": {} }
>>  ##
>> -{ 'command': 'device_del', 'data': {'id': 'str'} }
>> +{ 'command': 'device_del', 'data': {'id': 'str', '*force': 'bool'} }
>>  
>>  ##
>>  # @DEVICE_DELETED:
>> diff --git a/system/qdev-monitor.c b/system/qdev-monitor.c
>> index fa3cae246b..ca10a25c46 100644
>> --- a/system/qdev-monitor.c
>> +++ b/system/qdev-monitor.c
>> @@ -956,11 +956,13 @@ void qdev_unplug(DeviceState *dev, bool force, Error **errp)
>>      error_propagate(errp, local_err);
>>  }
> 
> With regards,
> Daniel
> -- 
> |: https://urldefense.com/v3/__https://berrange.com__;!!ACWV5N9M2RV99hQ!
> Np_bt2bKAVenBs32tlW80kBTAEEBD9uY3zjXBII-WZ6Yj3TRV1O5I7WsA70hJm0q3YdPBwu-
> DL2qAJ3UvkOW$ <https://urldefense.com/v3/__https://berrange.com__;!!ACWV5N9M2RV99hQ!Np_bt2bKAVenBs32tlW80kBTAEEBD9uY3zjXBII-WZ6Yj3TRV1O5I7WsA70hJm0q3YdPBwu-DL2qAJ3UvkOW$>       ~~        https://urldefense.com/v3/__https://hachyderm.io/@berrange__;!!ACWV5N9M2RV99hQ!
> Np_bt2bKAVenBs32tlW80kBTAEEBD9uY3zjXBII-WZ6Yj3TRV1O5I7WsA70hJm0q3YdPBwu-
> DL2qADJDqpwa$ <https://urldefense.com/v3/__https://hachyderm.io/@berrange__;!!ACWV5N9M2RV99hQ!Np_bt2bKAVenBs32tlW80kBTAEEBD9uY3zjXBII-WZ6Yj3TRV1O5I7WsA70hJm0q3YdPBwu-DL2qADJDqpwa$> :|
> |: https://urldefense.com/v3/__https://libvirt.org__;!!ACWV5N9M2RV99hQ!
> Np_bt2bKAVenBs32tlW80kBTAEEBD9uY3zjXBII-WZ6Yj3TRV1O5I7WsA70hJm0q3YdPBwu-
> DL2qAPWCfpLe$ <https://urldefense.com/v3/__https://libvirt.org__;!!ACWV5N9M2RV99hQ!Np_bt2bKAVenBs32tlW80kBTAEEBD9uY3zjXBII-WZ6Yj3TRV1O5I7WsA70hJm0q3YdPBwu-DL2qAPWCfpLe$>          ~~          https://urldefense.com/v3/__https://entangle-photo.org__;!!ACWV5N9M2RV99hQ!
> Np_bt2bKAVenBs32tlW80kBTAEEBD9uY3zjXBII-WZ6Yj3TRV1O5I7WsA70hJm0q3YdPBwu-
> DL2qAN7fX7fq$ <https://urldefense.com/v3/__https://entangle-photo.org__;!!ACWV5N9M2RV99hQ!Np_bt2bKAVenBs32tlW80kBTAEEBD9uY3zjXBII-WZ6Yj3TRV1O5I7WsA70hJm0q3YdPBwu-DL2qAN7fX7fq$> :|
> |: https://urldefense.com/v3/__https://pixelfed.art/berrange__;!!ACWV5N9M2RV99hQ!
> Np_bt2bKAVenBs32tlW80kBTAEEBD9uY3zjXBII-WZ6Yj3TRV1O5I7WsA70hJm0q3YdPBwu-
> DL2qAH19vj8f$ <https://urldefense.com/v3/__https://pixelfed.art/berrange__;!!ACWV5N9M2RV99hQ!Np_bt2bKAVenBs32tlW80kBTAEEBD9uY3zjXBII-WZ6Yj3TRV1O5I7WsA70hJm0q3YdPBwu-DL2qAH19vj8f$>   ~~    https://urldefense.com/v3/__https://fstop138.berrange.com__;!!ACWV5N9M2RV99hQ!
> Np_bt2bKAVenBs32tlW80kBTAEEBD9uY3zjXBII-WZ6Yj3TRV1O5I7WsA70hJm0q3YdPBwu-
> DL2qAJz0R0XI$ <https://urldefense.com/v3/__https://fstop138.berrange.com__;!!ACWV5N9M2RV99hQ!Np_bt2bKAVenBs32tlW80kBTAEEBD9uY3zjXBII-WZ6Yj3TRV1O5I7WsA70hJm0q3YdPBwu-DL2qAJz0R0XI$> :|
> 

Re: [PATCH 0/8] Force detach PCI devices for ACPI-based and PCIe native hot-unplug

[Dongli Zhang]

On Mon, Aug 24, 2026 7:44:49AM -0700, Daniel P. Berrangé wrote:
> On Sun, Aug 23, 2026 at 06:13:30PM -0700, Dongli Zhang wrote:
>> Hot-unplugging a PCI device can require cooperation from the guest. For
>> ACPI PCI hotplug, QEMU notifies the guest through ACPI and the guest
>> eventually writes the ACPI PCI eject register. For PCIe native hotplug,
>> QEMU notifies the guest through the PCIe hotplug mechanism and waits for
>> the slot unplug flow to complete. Only after that completion does QEMU
>> unrealize the device and emit DEVICE_DELETED.
>> 
>> This can leave a device stuck in the unplug pending state when the guest
>> does not cooperate. Examples include:
>> 
>> 1. The guest has panicked, or the relevant ACPI/PCI hotplug driver is
>> unavailable.
>> 
>> 2. The guest is stalled and cannot handle the hot-unplug event. For
>> example, stalling the Linux [irq/9-acpi] kernel thread can reproduce this
>> for ACPI-based hot-unplug.
>> 
>> 3. The device was attached to a slot that the guest cannot use. For
>> example, a pcie-root-port only supports slot 0. If a device is added to a
>> non-zero slot below a pcie-root-port, the guest may never discover the
>> device and therefore may never complete the unplug request.
>> 
>> The non-zero slot case has also been discussed in:
>> 
>> hw/pci: warn when PCIe device is plugged into non-zero slot of downstream port
>> https://gitlab.com/qemu-project/qemu/-/commit/
> ca92eb5defcf9d1c2106341744a73a03cf26e824
>> 
>> hw/pci: add comment to explain checking for available function 0 in pci hotplug
>> https://gitlab.com/qemu-project/qemu/-/
> commit/67d045a0ef5b9c5f871c3a1d87325a8a42d2b9d5
>> 
>> pci: don't skip function 0 occupancy verification for devfn auto assign
>> https://gitlab.com/qemu-project/qemu/-/commit/
> e228d62b4af29bca698ec57efdceb46f392f5444
>> 
>> For example, if root-port.1 is a pcie-root-port, the following command adds
>> a vhost-scsi-pci device to an invalid slot:
>> 
>> (qemu) device_add vhost-scsi-pci,id=scsi01,wwpn=naa.5001405324af0985,bus=root-port.1,addr=01.0
>> warning: PCI: slot 1 is not valid for vhost-scsi-pci, parent device only allows plugging into slot 0.
> 
> This rather looks like it should be a fatal error, not a mere warning.
> 
> If I follow the commit ca92eb5def it links to https://bugzilla.redhat.com/show_bug.cgi?id=2128929
> which states that this configuration is going to lead to a crash in
> QEMU on guest OS shutdown. IMHO that crash is sufficient to justify
> making this a fatal error.
> 
> If we actually wanted this to remain a warning, then that shutdown
> crash would need to be fixed.
> 

Thank you very much!

I see that the issue has been fixed. The ticket mentions the following.

"What I am observing is that it seems when the slot ID != 0, the guest OS seems
to ignore this and we never seem to hit ich9_pm_device_unplug_cb()."

Based on my experience and evaluation, ACPI-based hotplug is more likely to
encounter an issue where the guest VM does not respond to an unplug operation.

Thank you very much!

Dongli Zhang

Re: [PATCH 0/8] Force detach PCI devices for ACPI-based and PCIe native hot-unplug

[Igor Mammedov]

On Wed, 26 Aug 2026 09:15:47 -0700
Dongli Zhang <dongli.zhang@oracle.com> wrote:

> On Mon, Aug 24, 2026 7:44:49AM -0700, Daniel P. Berrangé wrote:
> > On Sun, Aug 23, 2026 at 06:13:30PM -0700, Dongli Zhang wrote:  
> >> Hot-unplugging a PCI device can require cooperation from the guest. For
> >> ACPI PCI hotplug, QEMU notifies the guest through ACPI and the guest
> >> eventually writes the ACPI PCI eject register. For PCIe native hotplug,
> >> QEMU notifies the guest through the PCIe hotplug mechanism and waits for
> >> the slot unplug flow to complete. Only after that completion does QEMU
> >> unrealize the device and emit DEVICE_DELETED.
> >> 
> >> This can leave a device stuck in the unplug pending state when the guest
> >> does not cooperate. Examples include:
> >> 
> >> 1. The guest has panicked, or the relevant ACPI/PCI hotplug driver is
> >> unavailable.
> >> 
> >> 2. The guest is stalled and cannot handle the hot-unplug event. For
> >> example, stalling the Linux [irq/9-acpi] kernel thread can reproduce this
> >> for ACPI-based hot-unplug.
> >> 
> >> 3. The device was attached to a slot that the guest cannot use. For
> >> example, a pcie-root-port only supports slot 0. If a device is added to a
> >> non-zero slot below a pcie-root-port, the guest may never discover the
> >> device and therefore may never complete the unplug request.

all of above is actually expected, no (functioning) driver => no hotplug/unplug.
it's the guest problem. Once device it exposed to guest its life-cycle
not longer owned by QEMU.

That's what one would see in real hw as well, you press eject button
but it will not do anything if OS doesn't process it.
also see comment at the end.

> >> 
> >> The non-zero slot case has also been discussed in:
> >> 
> >> hw/pci: warn when PCIe device is plugged into non-zero slot of downstream port
> >> https://gitlab.com/qemu-project/qemu/-/commit/  
> > ca92eb5defcf9d1c2106341744a73a03cf26e824  
> >> 
> >> hw/pci: add comment to explain checking for available function 0 in pci hotplug
> >> https://gitlab.com/qemu-project/qemu/-/  
> > commit/67d045a0ef5b9c5f871c3a1d87325a8a42d2b9d5  
> >> 
> >> pci: don't skip function 0 occupancy verification for devfn auto assign
> >> https://gitlab.com/qemu-project/qemu/-/commit/  
> > e228d62b4af29bca698ec57efdceb46f392f5444  
> >> 
> >> For example, if root-port.1 is a pcie-root-port, the following command adds
> >> a vhost-scsi-pci device to an invalid slot:
> >> 
> >> (qemu) device_add vhost-scsi-pci,id=scsi01,wwpn=naa.5001405324af0985,bus=root-port.1,addr=01.0
> >> warning: PCI: slot 1 is not valid for vhost-scsi-pci, parent device only allows plugging into slot 0.  
> > 
> > This rather looks like it should be a fatal error, not a mere warning.
> > 
> > If I follow the commit ca92eb5def it links to https://bugzilla.redhat.com/show_bug.cgi?id=2128929
> > which states that this configuration is going to lead to a crash in
> > QEMU on guest OS shutdown. IMHO that crash is sufficient to justify
> > making this a fatal error.
> > 
> > If we actually wanted this to remain a warning, then that shutdown
> > crash would need to be fixed.
> >   
> 
> Thank you very much!
> 
> I see that the issue has been fixed. The ticket mentions the following.
> 
> "What I am observing is that it seems when the slot ID != 0, the guest OS seems
> to ignore this and we never seem to hit ich9_pm_device_unplug_cb()."
> 
> Based on my experience and evaluation, ACPI-based hotplug is more likely to
> encounter an issue where the guest VM does not respond to an unplug operation.

I'm not sure it's a good idea to delete device when guest still thinks it's there
(you can make guesses on QEMU side if it's in use, how useful those are is questionable).

as far as I know, ACPI hotplug has no notion of surprise removal (pls educate me if it's not the case),
so I wouldn't do what you are proposing here at all, it's basically asking for disaster to happen.
And all this is basically for dealing with abused qemu flexibility.

Please (re)formulate usecase and make it more clear as what is eludes me
no matter how many times i've read this cover letter.

On positive note:

What you can try to implement is native PCI-E support for surprise removal.
How hard that would be I don't know. And I would well expect if one deviates from
real hw expectations/configs (such as not 0 slot/partial func removal),
one would quickly stumble upon issues as  that's not what what vendors write/test
drivers for.

Even if it's not likely to be used in practice (guest still might not support it),
it may serve as test-bed for guest drivers.

> Thank you very much!
> 
> Dongli Zhang
> 

Re: [PATCH 0/8] Force detach PCI devices for ACPI-based and PCIe native hot-unplug

[Michael S. Tsirkin]

On Thu, Sep 03, 2026 at 05:28:35PM +0200, Igor Mammedov wrote:
> On Wed, 26 Aug 2026 09:15:47 -0700
> Dongli Zhang <dongli.zhang@oracle.com> wrote:
> 
> > On Mon, Aug 24, 2026 7:44:49AM -0700, Daniel P. Berrangé wrote:
> > > On Sun, Aug 23, 2026 at 06:13:30PM -0700, Dongli Zhang wrote:  
> > >> Hot-unplugging a PCI device can require cooperation from the guest. For
> > >> ACPI PCI hotplug, QEMU notifies the guest through ACPI and the guest
> > >> eventually writes the ACPI PCI eject register. For PCIe native hotplug,
> > >> QEMU notifies the guest through the PCIe hotplug mechanism and waits for
> > >> the slot unplug flow to complete. Only after that completion does QEMU
> > >> unrealize the device and emit DEVICE_DELETED.
> > >> 
> > >> This can leave a device stuck in the unplug pending state when the guest
> > >> does not cooperate. Examples include:
> > >> 
> > >> 1. The guest has panicked, or the relevant ACPI/PCI hotplug driver is
> > >> unavailable.
> > >> 
> > >> 2. The guest is stalled and cannot handle the hot-unplug event. For
> > >> example, stalling the Linux [irq/9-acpi] kernel thread can reproduce this
> > >> for ACPI-based hot-unplug.
> > >> 
> > >> 3. The device was attached to a slot that the guest cannot use. For
> > >> example, a pcie-root-port only supports slot 0. If a device is added to a
> > >> non-zero slot below a pcie-root-port, the guest may never discover the
> > >> device and therefore may never complete the unplug request.
> 
> all of above is actually expected, no (functioning) driver => no hotplug/unplug.
> it's the guest problem. Once device it exposed to guest its life-cycle
> not longer owned by QEMU.
> 
> That's what one would see in real hw as well, you press eject button
> but it will not do anything if OS doesn't process it.
> also see comment at the end.
> 
> > >> 
> > >> The non-zero slot case has also been discussed in:
> > >> 
> > >> hw/pci: warn when PCIe device is plugged into non-zero slot of downstream port
> > >> https://gitlab.com/qemu-project/qemu/-/commit/  
> > > ca92eb5defcf9d1c2106341744a73a03cf26e824  
> > >> 
> > >> hw/pci: add comment to explain checking for available function 0 in pci hotplug
> > >> https://gitlab.com/qemu-project/qemu/-/  
> > > commit/67d045a0ef5b9c5f871c3a1d87325a8a42d2b9d5  
> > >> 
> > >> pci: don't skip function 0 occupancy verification for devfn auto assign
> > >> https://gitlab.com/qemu-project/qemu/-/commit/  
> > > e228d62b4af29bca698ec57efdceb46f392f5444  
> > >> 
> > >> For example, if root-port.1 is a pcie-root-port, the following command adds
> > >> a vhost-scsi-pci device to an invalid slot:
> > >> 
> > >> (qemu) device_add vhost-scsi-pci,id=scsi01,wwpn=naa.5001405324af0985,bus=root-port.1,addr=01.0
> > >> warning: PCI: slot 1 is not valid for vhost-scsi-pci, parent device only allows plugging into slot 0.  
> > > 
> > > This rather looks like it should be a fatal error, not a mere warning.
> > > 
> > > If I follow the commit ca92eb5def it links to https://bugzilla.redhat.com/show_bug.cgi?id=2128929
> > > which states that this configuration is going to lead to a crash in
> > > QEMU on guest OS shutdown. IMHO that crash is sufficient to justify
> > > making this a fatal error.
> > > 
> > > If we actually wanted this to remain a warning, then that shutdown
> > > crash would need to be fixed.
> > >   
> > 
> > Thank you very much!
> > 
> > I see that the issue has been fixed. The ticket mentions the following.
> > 
> > "What I am observing is that it seems when the slot ID != 0, the guest OS seems
> > to ignore this and we never seem to hit ich9_pm_device_unplug_cb()."
> > 
> > Based on my experience and evaluation, ACPI-based hotplug is more likely to
> > encounter an issue where the guest VM does not respond to an unplug operation.
> 
> I'm not sure it's a good idea to delete device when guest still thinks it's there
> (you can make guesses on QEMU side if it's in use, how useful those are is questionable).
> 
> as far as I know, ACPI hotplug has no notion of surprise removal (pls educate me if it's not the case),

why would it not?

how do you think you can pull a laptop out of a dock?
I expect bus check + _STA and config space saying it is gone
will do exactly that.


Here's linux code:
static void acpiphp_check_bridge(struct acpiphp_bridge *bridge)
{       
        struct acpiphp_slot *slot;

        /* Bail out if the bridge is going away. */
        if (bridge->is_going_away)
                return;

        if (bridge->pci_dev)
                pm_runtime_get_sync(&bridge->pci_dev->dev);
        
        list_for_each_entry(slot, &bridge->slots, node) {
                struct pci_bus *bus = slot->bus;
                struct pci_dev *dev, *tmp;

                if (slot_no_hotplug(slot)) {
                        ; /* do nothing */
                } else if (device_status_valid(get_slot_status(slot))) {
                        /* remove stale devices if any */
                        list_for_each_entry_safe_reverse(dev, tmp,
                                                         &bus->devices, bus_list)
                                if (PCI_SLOT(dev->devfn) == slot->device)
                                        trim_stale_devices(dev);

                        /* configure all functions */
                        enable_slot(slot, true);
                } else {
                        disable_slot(slot);
                }
        }

        if (bridge->pci_dev)
                pm_runtime_put(&bridge->pci_dev->dev);
}


so weirdly it wants bus check on a parent bus, otherwise it will
not trim devices?  probably a bug, but easy to work around.


> so I wouldn't do what you are proposing here at all, it's basically asking for disaster to happen.
> And all this is basically for dealing with abused qemu flexibility.
> 
> Please (re)formulate usecase and make it more clear as what is eludes me
> no matter how many times i've read this cover letter.
> 
> On positive note:
> 
> What you can try to implement is native PCI-E support for surprise removal.
> How hard that would be I don't know. And I would well expect if one deviates from
> real hw expectations/configs (such as not 0 slot/partial func removal),
> one would quickly stumble upon issues as  that's not what what vendors write/test
> drivers for.
> 
> Even if it's not likely to be used in practice (guest still might not support it),
> it may serve as test-bed for guest drivers.
> 
> > Thank you very much!
> > 
> > Dongli Zhang
> > 

Re: [PATCH 0/8] Force detach PCI devices for ACPI-based and PCIe native hot-unplug

[Igor Mammedov]

On Thu, 3 Sep 2026 16:17:20 -0400
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Thu, Sep 03, 2026 at 05:28:35PM +0200, Igor Mammedov wrote:
> > On Wed, 26 Aug 2026 09:15:47 -0700
> > Dongli Zhang <dongli.zhang@oracle.com> wrote:
> >   
> > > On Mon, Aug 24, 2026 7:44:49AM -0700, Daniel P. Berrangé wrote:  
> > > > On Sun, Aug 23, 2026 at 06:13:30PM -0700, Dongli Zhang wrote:    
> > > >> Hot-unplugging a PCI device can require cooperation from the guest. For
> > > >> ACPI PCI hotplug, QEMU notifies the guest through ACPI and the guest
> > > >> eventually writes the ACPI PCI eject register. For PCIe native hotplug,
> > > >> QEMU notifies the guest through the PCIe hotplug mechanism and waits for
> > > >> the slot unplug flow to complete. Only after that completion does QEMU
> > > >> unrealize the device and emit DEVICE_DELETED.
> > > >> 
> > > >> This can leave a device stuck in the unplug pending state when the guest
> > > >> does not cooperate. Examples include:
> > > >> 
> > > >> 1. The guest has panicked, or the relevant ACPI/PCI hotplug driver is
> > > >> unavailable.
> > > >> 
> > > >> 2. The guest is stalled and cannot handle the hot-unplug event. For
> > > >> example, stalling the Linux [irq/9-acpi] kernel thread can reproduce this
> > > >> for ACPI-based hot-unplug.
> > > >> 
> > > >> 3. The device was attached to a slot that the guest cannot use. For
> > > >> example, a pcie-root-port only supports slot 0. If a device is added to a
> > > >> non-zero slot below a pcie-root-port, the guest may never discover the
> > > >> device and therefore may never complete the unplug request.  
> > 
> > all of above is actually expected, no (functioning) driver => no hotplug/unplug.
> > it's the guest problem. Once device it exposed to guest its life-cycle
> > not longer owned by QEMU.
> > 
> > That's what one would see in real hw as well, you press eject button
> > but it will not do anything if OS doesn't process it.
> > also see comment at the end.
> >   
> > > >> 
> > > >> The non-zero slot case has also been discussed in:
> > > >> 
> > > >> hw/pci: warn when PCIe device is plugged into non-zero slot of downstream port
> > > >> https://gitlab.com/qemu-project/qemu/-/commit/    
> > > > ca92eb5defcf9d1c2106341744a73a03cf26e824    
> > > >> 
> > > >> hw/pci: add comment to explain checking for available function 0 in pci hotplug
> > > >> https://gitlab.com/qemu-project/qemu/-/    
> > > > commit/67d045a0ef5b9c5f871c3a1d87325a8a42d2b9d5    
> > > >> 
> > > >> pci: don't skip function 0 occupancy verification for devfn auto assign
> > > >> https://gitlab.com/qemu-project/qemu/-/commit/    
> > > > e228d62b4af29bca698ec57efdceb46f392f5444    
> > > >> 
> > > >> For example, if root-port.1 is a pcie-root-port, the following command adds
> > > >> a vhost-scsi-pci device to an invalid slot:
> > > >> 
> > > >> (qemu) device_add vhost-scsi-pci,id=scsi01,wwpn=naa.5001405324af0985,bus=root-port.1,addr=01.0
> > > >> warning: PCI: slot 1 is not valid for vhost-scsi-pci, parent device only allows plugging into slot 0.    
> > > > 
> > > > This rather looks like it should be a fatal error, not a mere warning.
> > > > 
> > > > If I follow the commit ca92eb5def it links to https://bugzilla.redhat.com/show_bug.cgi?id=2128929
> > > > which states that this configuration is going to lead to a crash in
> > > > QEMU on guest OS shutdown. IMHO that crash is sufficient to justify
> > > > making this a fatal error.
> > > > 
> > > > If we actually wanted this to remain a warning, then that shutdown
> > > > crash would need to be fixed.
> > > >     
> > > 
> > > Thank you very much!
> > > 
> > > I see that the issue has been fixed. The ticket mentions the following.
> > > 
> > > "What I am observing is that it seems when the slot ID != 0, the guest OS seems
> > > to ignore this and we never seem to hit ich9_pm_device_unplug_cb()."
> > > 
> > > Based on my experience and evaluation, ACPI-based hotplug is more likely to
> > > encounter an issue where the guest VM does not respond to an unplug operation.  
> > 
> > I'm not sure it's a good idea to delete device when guest still thinks it's there
> > (you can make guesses on QEMU side if it's in use, how useful those are is questionable).
> > 
> > as far as I know, ACPI hotplug has no notion of surprise removal (pls educate me if it's not the case),  
> 
> why would it not?
> 
> how do you think you can pull a laptop out of a dock?
> I expect bus check + _STA and config space saying it is gone
> will do exactly that.
> 
> 
> Here's linux code:
> static void acpiphp_check_bridge(struct acpiphp_bridge *bridge)
> {       
>         struct acpiphp_slot *slot;
> 
>         /* Bail out if the bridge is going away. */
>         if (bridge->is_going_away)
>                 return;
> 
>         if (bridge->pci_dev)
>                 pm_runtime_get_sync(&bridge->pci_dev->dev);
>         
>         list_for_each_entry(slot, &bridge->slots, node) {
>                 struct pci_bus *bus = slot->bus;
>                 struct pci_dev *dev, *tmp;
> 
>                 if (slot_no_hotplug(slot)) {
>                         ; /* do nothing */
>                 } else if (device_status_valid(get_slot_status(slot))) {
>                         /* remove stale devices if any */
>                         list_for_each_entry_safe_reverse(dev, tmp,
>                                                          &bus->devices, bus_list)
>                                 if (PCI_SLOT(dev->devfn) == slot->device)
>                                         trim_stale_devices(dev);
> 
>                         /* configure all functions */
>                         enable_slot(slot, true);
>                 } else {
>                         disable_slot(slot);
>                 }
>         }
> 
>         if (bridge->pci_dev)
>                 pm_runtime_put(&bridge->pci_dev->dev);
> }
> 
> 
> so weirdly it wants bus check on a parent bus, otherwise it will
> not trim devices?  probably a bug, but easy to work around.

Modern docks would use native pcie surprise removal path.

As for ACPI, my old laptop, had an unlock button => _LCK
and that relied on OS processing ACPI events, not so surprise.

There might have been ACPI/hybrid docks that did surprise removal,
but then one need to find one and model after that instead of 
just blanket force removal. (likely out come would a doc device
support only, not an arbitrary device removal)

(not the case described in this series, though. hence my request to clarify usecase)

from what I see in spec there is _RMV method that says that device
supports surprise removal that can be used for devices that support it.
However I would hesitate very much to blank apply it to every PCI device.
(it's not even realistic to ask for proving safe tear down across various
drivers and OSes/versions)

Rather than a knee jerk treatment of misconfig consequences,
I'd rather see patches to prevent misconfig in the 1st place
(subj to deprecation but doable).

As for the cases where OS mis-behaves (apcihp thread starvation,...),
fixing guest to follow hotplug contract is a proper place to do it.

On QEMU side we have it covered as well. If unplug was not processed,
mgmt is free to repeat action.

 
> > so I wouldn't do what you are proposing here at all, it's basically asking for disaster to happen.
> > And all this is basically for dealing with abused qemu flexibility.
> > 
> > Please (re)formulate usecase and make it more clear as what is eludes me
> > no matter how many times i've read this cover letter.
> > 
> > On positive note:
> > 
> > What you can try to implement is native PCI-E support for surprise removal.
> > How hard that would be I don't know. And I would well expect if one deviates from
> > real hw expectations/configs (such as not 0 slot/partial func removal),
> > one would quickly stumble upon issues as  that's not what what vendors write/test
> > drivers for.
> > 
> > Even if it's not likely to be used in practice (guest still might not support it),
> > it may serve as test-bed for guest drivers.
> >   
> > > Thank you very much!
> > > 
> > > Dongli Zhang
> > >   
> 

Re: [PATCH 0/8] Force detach PCI devices for ACPI-based and PCIe native hot-unplug

[Michael S. Tsirkin]

On Fri, Sep 04, 2026 at 01:07:09PM +0200, Igor Mammedov wrote:
> On Thu, 3 Sep 2026 16:17:20 -0400
> "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > On Thu, Sep 03, 2026 at 05:28:35PM +0200, Igor Mammedov wrote:
> > > On Wed, 26 Aug 2026 09:15:47 -0700
> > > Dongli Zhang <dongli.zhang@oracle.com> wrote:
> > >   
> > > > On Mon, Aug 24, 2026 7:44:49AM -0700, Daniel P. Berrangé wrote:  
> > > > > On Sun, Aug 23, 2026 at 06:13:30PM -0700, Dongli Zhang wrote:    
> > > > >> Hot-unplugging a PCI device can require cooperation from the guest. For
> > > > >> ACPI PCI hotplug, QEMU notifies the guest through ACPI and the guest
> > > > >> eventually writes the ACPI PCI eject register. For PCIe native hotplug,
> > > > >> QEMU notifies the guest through the PCIe hotplug mechanism and waits for
> > > > >> the slot unplug flow to complete. Only after that completion does QEMU
> > > > >> unrealize the device and emit DEVICE_DELETED.
> > > > >> 
> > > > >> This can leave a device stuck in the unplug pending state when the guest
> > > > >> does not cooperate. Examples include:
> > > > >> 
> > > > >> 1. The guest has panicked, or the relevant ACPI/PCI hotplug driver is
> > > > >> unavailable.
> > > > >> 
> > > > >> 2. The guest is stalled and cannot handle the hot-unplug event. For
> > > > >> example, stalling the Linux [irq/9-acpi] kernel thread can reproduce this
> > > > >> for ACPI-based hot-unplug.
> > > > >> 
> > > > >> 3. The device was attached to a slot that the guest cannot use. For
> > > > >> example, a pcie-root-port only supports slot 0. If a device is added to a
> > > > >> non-zero slot below a pcie-root-port, the guest may never discover the
> > > > >> device and therefore may never complete the unplug request.  
> > > 
> > > all of above is actually expected, no (functioning) driver => no hotplug/unplug.
> > > it's the guest problem. Once device it exposed to guest its life-cycle
> > > not longer owned by QEMU.
> > > 
> > > That's what one would see in real hw as well, you press eject button
> > > but it will not do anything if OS doesn't process it.
> > > also see comment at the end.
> > >   
> > > > >> 
> > > > >> The non-zero slot case has also been discussed in:
> > > > >> 
> > > > >> hw/pci: warn when PCIe device is plugged into non-zero slot of downstream port
> > > > >> https://gitlab.com/qemu-project/qemu/-/commit/    
> > > > > ca92eb5defcf9d1c2106341744a73a03cf26e824    
> > > > >> 
> > > > >> hw/pci: add comment to explain checking for available function 0 in pci hotplug
> > > > >> https://gitlab.com/qemu-project/qemu/-/    
> > > > > commit/67d045a0ef5b9c5f871c3a1d87325a8a42d2b9d5    
> > > > >> 
> > > > >> pci: don't skip function 0 occupancy verification for devfn auto assign
> > > > >> https://gitlab.com/qemu-project/qemu/-/commit/    
> > > > > e228d62b4af29bca698ec57efdceb46f392f5444    
> > > > >> 
> > > > >> For example, if root-port.1 is a pcie-root-port, the following command adds
> > > > >> a vhost-scsi-pci device to an invalid slot:
> > > > >> 
> > > > >> (qemu) device_add vhost-scsi-pci,id=scsi01,wwpn=naa.5001405324af0985,bus=root-port.1,addr=01.0
> > > > >> warning: PCI: slot 1 is not valid for vhost-scsi-pci, parent device only allows plugging into slot 0.    
> > > > > 
> > > > > This rather looks like it should be a fatal error, not a mere warning.
> > > > > 
> > > > > If I follow the commit ca92eb5def it links to https://bugzilla.redhat.com/show_bug.cgi?id=2128929
> > > > > which states that this configuration is going to lead to a crash in
> > > > > QEMU on guest OS shutdown. IMHO that crash is sufficient to justify
> > > > > making this a fatal error.
> > > > > 
> > > > > If we actually wanted this to remain a warning, then that shutdown
> > > > > crash would need to be fixed.
> > > > >     
> > > > 
> > > > Thank you very much!
> > > > 
> > > > I see that the issue has been fixed. The ticket mentions the following.
> > > > 
> > > > "What I am observing is that it seems when the slot ID != 0, the guest OS seems
> > > > to ignore this and we never seem to hit ich9_pm_device_unplug_cb()."
> > > > 
> > > > Based on my experience and evaluation, ACPI-based hotplug is more likely to
> > > > encounter an issue where the guest VM does not respond to an unplug operation.  
> > > 
> > > I'm not sure it's a good idea to delete device when guest still thinks it's there
> > > (you can make guesses on QEMU side if it's in use, how useful those are is questionable).
> > > 
> > > as far as I know, ACPI hotplug has no notion of surprise removal (pls educate me if it's not the case),  
> > 
> > why would it not?
> > 
> > how do you think you can pull a laptop out of a dock?
> > I expect bus check + _STA and config space saying it is gone
> > will do exactly that.
> > 
> > 
> > Here's linux code:
> > static void acpiphp_check_bridge(struct acpiphp_bridge *bridge)
> > {       
> >         struct acpiphp_slot *slot;
> > 
> >         /* Bail out if the bridge is going away. */
> >         if (bridge->is_going_away)
> >                 return;
> > 
> >         if (bridge->pci_dev)
> >                 pm_runtime_get_sync(&bridge->pci_dev->dev);
> >         
> >         list_for_each_entry(slot, &bridge->slots, node) {
> >                 struct pci_bus *bus = slot->bus;
> >                 struct pci_dev *dev, *tmp;
> > 
> >                 if (slot_no_hotplug(slot)) {
> >                         ; /* do nothing */
> >                 } else if (device_status_valid(get_slot_status(slot))) {
> >                         /* remove stale devices if any */
> >                         list_for_each_entry_safe_reverse(dev, tmp,
> >                                                          &bus->devices, bus_list)
> >                                 if (PCI_SLOT(dev->devfn) == slot->device)
> >                                         trim_stale_devices(dev);
> > 
> >                         /* configure all functions */
> >                         enable_slot(slot, true);
> >                 } else {
> >                         disable_slot(slot);
> >                 }
> >         }
> > 
> >         if (bridge->pci_dev)
> >                 pm_runtime_put(&bridge->pci_dev->dev);
> > }
> > 
> > 
> > so weirdly it wants bus check on a parent bus, otherwise it will
> > not trim devices?  probably a bug, but easy to work around.
> 
> Modern docks would use native pcie surprise removal path.
> 
> As for ACPI, my old laptop, had an unlock button => _LCK
> and that relied on OS processing ACPI events, not so surprise.
> 
> There might have been ACPI/hybrid docks that did surprise removal,
> but then one need to find one and model after that instead of 
> just blanket force removal. (likely out come would a doc device
> support only, not an arbitrary device removal)
> 
> (not the case described in this series, though. hence my request to clarify usecase)
> 
> from what I see in spec there is _RMV method that says that device
> supports surprise removal that can be used for devices that support it.
> However I would hesitate very much to blank apply it to every PCI device.
> (it's not even realistic to ask for proving safe tear down across various
> drivers and OSes/versions)
> 
> Rather than a knee jerk treatment of misconfig consequences,
> I'd rather see patches to prevent misconfig in the 1st place
> (subj to deprecation but doable).
> 
> As for the cases where OS mis-behaves (apcihp thread starvation,...),
> fixing guest to follow hotplug contract is a proper place to do it.
> 
> On QEMU side we have it covered as well. If unplug was not processed,
> mgmt is free to repeat action.


Sorry if I am unclear. I just meant that it looks like we
can support surprise removal with ACPI just by reporting
bus check events on the parent.


>  
> > > so I wouldn't do what you are proposing here at all, it's basically asking for disaster to happen.
> > > And all this is basically for dealing with abused qemu flexibility.
> > > 
> > > Please (re)formulate usecase and make it more clear as what is eludes me
> > > no matter how many times i've read this cover letter.
> > > 
> > > On positive note:
> > > 
> > > What you can try to implement is native PCI-E support for surprise removal.
> > > How hard that would be I don't know. And I would well expect if one deviates from
> > > real hw expectations/configs (such as not 0 slot/partial func removal),
> > > one would quickly stumble upon issues as  that's not what what vendors write/test
> > > drivers for.
> > > 
> > > Even if it's not likely to be used in practice (guest still might not support it),
> > > it may serve as test-bed for guest drivers.
> > >   
> > > > Thank you very much!
> > > > 
> > > > Dongli Zhang
> > > >   
> > 

Re: [PATCH 0/8] Force detach PCI devices for ACPI-based and PCIe native hot-unplug

[Igor Mammedov]

On Fri, 4 Sep 2026 07:22:10 -0400
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Fri, Sep 04, 2026 at 01:07:09PM +0200, Igor Mammedov wrote:
> > On Thu, 3 Sep 2026 16:17:20 -0400
> > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >   
> > > On Thu, Sep 03, 2026 at 05:28:35PM +0200, Igor Mammedov wrote:  
> > > > On Wed, 26 Aug 2026 09:15:47 -0700
> > > > Dongli Zhang <dongli.zhang@oracle.com> wrote:
> > > >     
> > > > > On Mon, Aug 24, 2026 7:44:49AM -0700, Daniel P. Berrangé wrote:    
> > > > > > On Sun, Aug 23, 2026 at 06:13:30PM -0700, Dongli Zhang wrote:      
> > > > > >> Hot-unplugging a PCI device can require cooperation from the guest. For
> > > > > >> ACPI PCI hotplug, QEMU notifies the guest through ACPI and the guest
> > > > > >> eventually writes the ACPI PCI eject register. For PCIe native hotplug,
> > > > > >> QEMU notifies the guest through the PCIe hotplug mechanism and waits for
> > > > > >> the slot unplug flow to complete. Only after that completion does QEMU
> > > > > >> unrealize the device and emit DEVICE_DELETED.
> > > > > >> 
> > > > > >> This can leave a device stuck in the unplug pending state when the guest
> > > > > >> does not cooperate. Examples include:
> > > > > >> 
> > > > > >> 1. The guest has panicked, or the relevant ACPI/PCI hotplug driver is
> > > > > >> unavailable.
> > > > > >> 
> > > > > >> 2. The guest is stalled and cannot handle the hot-unplug event. For
> > > > > >> example, stalling the Linux [irq/9-acpi] kernel thread can reproduce this
> > > > > >> for ACPI-based hot-unplug.
> > > > > >> 
> > > > > >> 3. The device was attached to a slot that the guest cannot use. For
> > > > > >> example, a pcie-root-port only supports slot 0. If a device is added to a
> > > > > >> non-zero slot below a pcie-root-port, the guest may never discover the
> > > > > >> device and therefore may never complete the unplug request.    
> > > > 
> > > > all of above is actually expected, no (functioning) driver => no hotplug/unplug.
> > > > it's the guest problem. Once device it exposed to guest its life-cycle
> > > > not longer owned by QEMU.
> > > > 
> > > > That's what one would see in real hw as well, you press eject button
> > > > but it will not do anything if OS doesn't process it.
> > > > also see comment at the end.
> > > >     
> > > > > >> 
> > > > > >> The non-zero slot case has also been discussed in:
> > > > > >> 
> > > > > >> hw/pci: warn when PCIe device is plugged into non-zero slot of downstream port
> > > > > >> https://gitlab.com/qemu-project/qemu/-/commit/      
> > > > > > ca92eb5defcf9d1c2106341744a73a03cf26e824      
> > > > > >> 
> > > > > >> hw/pci: add comment to explain checking for available function 0 in pci hotplug
> > > > > >> https://gitlab.com/qemu-project/qemu/-/      
> > > > > > commit/67d045a0ef5b9c5f871c3a1d87325a8a42d2b9d5      
> > > > > >> 
> > > > > >> pci: don't skip function 0 occupancy verification for devfn auto assign
> > > > > >> https://gitlab.com/qemu-project/qemu/-/commit/      
> > > > > > e228d62b4af29bca698ec57efdceb46f392f5444      
> > > > > >> 
> > > > > >> For example, if root-port.1 is a pcie-root-port, the following command adds
> > > > > >> a vhost-scsi-pci device to an invalid slot:
> > > > > >> 
> > > > > >> (qemu) device_add vhost-scsi-pci,id=scsi01,wwpn=naa.5001405324af0985,bus=root-port.1,addr=01.0
> > > > > >> warning: PCI: slot 1 is not valid for vhost-scsi-pci, parent device only allows plugging into slot 0.      
> > > > > > 
> > > > > > This rather looks like it should be a fatal error, not a mere warning.
> > > > > > 
> > > > > > If I follow the commit ca92eb5def it links to https://bugzilla.redhat.com/show_bug.cgi?id=2128929
> > > > > > which states that this configuration is going to lead to a crash in
> > > > > > QEMU on guest OS shutdown. IMHO that crash is sufficient to justify
> > > > > > making this a fatal error.
> > > > > > 
> > > > > > If we actually wanted this to remain a warning, then that shutdown
> > > > > > crash would need to be fixed.
> > > > > >       
> > > > > 
> > > > > Thank you very much!
> > > > > 
> > > > > I see that the issue has been fixed. The ticket mentions the following.
> > > > > 
> > > > > "What I am observing is that it seems when the slot ID != 0, the guest OS seems
> > > > > to ignore this and we never seem to hit ich9_pm_device_unplug_cb()."
> > > > > 
> > > > > Based on my experience and evaluation, ACPI-based hotplug is more likely to
> > > > > encounter an issue where the guest VM does not respond to an unplug operation.    
> > > > 
> > > > I'm not sure it's a good idea to delete device when guest still thinks it's there
> > > > (you can make guesses on QEMU side if it's in use, how useful those are is questionable).
> > > > 
> > > > as far as I know, ACPI hotplug has no notion of surprise removal (pls educate me if it's not the case),    
> > > 
> > > why would it not?
> > > 
> > > how do you think you can pull a laptop out of a dock?
> > > I expect bus check + _STA and config space saying it is gone
> > > will do exactly that.
> > > 
> > > 
> > > Here's linux code:
> > > static void acpiphp_check_bridge(struct acpiphp_bridge *bridge)
> > > {       
> > >         struct acpiphp_slot *slot;
> > > 
> > >         /* Bail out if the bridge is going away. */
> > >         if (bridge->is_going_away)
> > >                 return;
> > > 
> > >         if (bridge->pci_dev)
> > >                 pm_runtime_get_sync(&bridge->pci_dev->dev);
> > >         
> > >         list_for_each_entry(slot, &bridge->slots, node) {
> > >                 struct pci_bus *bus = slot->bus;
> > >                 struct pci_dev *dev, *tmp;
> > > 
> > >                 if (slot_no_hotplug(slot)) {
> > >                         ; /* do nothing */
> > >                 } else if (device_status_valid(get_slot_status(slot))) {
> > >                         /* remove stale devices if any */
> > >                         list_for_each_entry_safe_reverse(dev, tmp,
> > >                                                          &bus->devices, bus_list)
> > >                                 if (PCI_SLOT(dev->devfn) == slot->device)
> > >                                         trim_stale_devices(dev);
> > > 
> > >                         /* configure all functions */
> > >                         enable_slot(slot, true);
> > >                 } else {
> > >                         disable_slot(slot);
> > >                 }
> > >         }
> > > 
> > >         if (bridge->pci_dev)
> > >                 pm_runtime_put(&bridge->pci_dev->dev);
> > > }
> > > 
> > > 
> > > so weirdly it wants bus check on a parent bus, otherwise it will
> > > not trim devices?  probably a bug, but easy to work around.  
> > 
> > Modern docks would use native pcie surprise removal path.
> > 
> > As for ACPI, my old laptop, had an unlock button => _LCK
> > and that relied on OS processing ACPI events, not so surprise.
> > 
> > There might have been ACPI/hybrid docks that did surprise removal,
> > but then one need to find one and model after that instead of 
> > just blanket force removal. (likely out come would a doc device
> > support only, not an arbitrary device removal)
> > 
> > (not the case described in this series, though. hence my request to clarify usecase)
> > 
> > from what I see in spec there is _RMV method that says that device
> > supports surprise removal that can be used for devices that support it.
> > However I would hesitate very much to blank apply it to every PCI device.
> > (it's not even realistic to ask for proving safe tear down across various
> > drivers and OSes/versions)
> > 
> > Rather than a knee jerk treatment of misconfig consequences,
> > I'd rather see patches to prevent misconfig in the 1st place
> > (subj to deprecation but doable).
> > 
> > As for the cases where OS mis-behaves (apcihp thread starvation,...),
> > fixing guest to follow hotplug contract is a proper place to do it.
> > 
> > On QEMU side we have it covered as well. If unplug was not processed,
> > mgmt is free to repeat action.  
> 
> 
> Sorry if I am unclear. I just meant that it looks like we
> can support surprise removal with ACPI just by reporting
> bus check events on the parent.

maybe, but that ain't SPECed and might be OS specific.

The way I've read the cover letter, that won't work for mentioned mis-config cases.
Also what would happen on bus-check 'cleanup' would be a lottery.
hence I'm for being safe here.

It's better to implement native PCIE surprise removal if that's really needed.

> > > > so I wouldn't do what you are proposing here at all, it's basically asking for disaster to happen.
> > > > And all this is basically for dealing with abused qemu flexibility.
> > > > 
> > > > Please (re)formulate usecase and make it more clear as what is eludes me
> > > > no matter how many times i've read this cover letter.
> > > > 
> > > > On positive note:
> > > > 
> > > > What you can try to implement is native PCI-E support for surprise removal.
> > > > How hard that would be I don't know. And I would well expect if one deviates from
> > > > real hw expectations/configs (such as not 0 slot/partial func removal),
> > > > one would quickly stumble upon issues as  that's not what what vendors write/test
> > > > drivers for.
> > > > 
> > > > Even if it's not likely to be used in practice (guest still might not support it),
> > > > it may serve as test-bed for guest drivers.
> > > >     
> > > > > Thank you very much!
> > > > > 
> > > > > Dongli Zhang
> > > > >     
> > >   
> 

Re: [PATCH 0/8] Force detach PCI devices for ACPI-based and PCIe native hot-unplug

[Dongli Zhang]

On Thu, Sep 3, 2026 8:28:35AM -0700, Igor Mammedov wrote:
> On Wed, 26 Aug 2026 09:15:47 -0700
> Dongli Zhang <dongli.zhang@oracle.com> wrote:
> 
>> On Mon, Aug 24, 2026 7:44:49AM -0700, Daniel P. Berrangé wrote:
>> > On Sun, Aug 23, 2026 at 06:13:30PM -0700, Dongli Zhang wrote:  
>> >> Hot-unplugging a PCI device can require cooperation from the guest. For
>> >> ACPI PCI hotplug, QEMU notifies the guest through ACPI and the guest
>> >> eventually writes the ACPI PCI eject register. For PCIe native hotplug,
>> >> QEMU notifies the guest through the PCIe hotplug mechanism and waits for
>> >> the slot unplug flow to complete. Only after that completion does QEMU
>> >> unrealize the device and emit DEVICE_DELETED.
>> >> 
>> >> This can leave a device stuck in the unplug pending state when the guest
>> >> does not cooperate. Examples include:
>> >> 
>> >> 1. The guest has panicked, or the relevant ACPI/PCI hotplug driver is
>> >> unavailable.
>> >> 
>> >> 2. The guest is stalled and cannot handle the hot-unplug event. For
>> >> example, stalling the Linux [irq/9-acpi] kernel thread can reproduce this
>> >> for ACPI-based hot-unplug.
>> >> 
>> >> 3. The device was attached to a slot that the guest cannot use. For
>> >> example, a pcie-root-port only supports slot 0. If a device is added to a
>> >> non-zero slot below a pcie-root-port, the guest may never discover the
>> >> device and therefore may never complete the unplug request.
> 
> all of above is actually expected, no (functioning) driver => no hotplug/unplug.
> it's the guest problem. Once device it exposed to guest its life-cycle
> not longer owned by QEMU.
> 
> That's what one would see in real hw as well, you press eject button
> but it will not do anything if OS doesn't process it.
> also see comment at the end.

Users are generally more tolerant of issues with real hardware.

In virtualization and cloud environments, PCI hotplug is more commonly used for
NICs and storage devices. Users are less tolerant of disruption or unexpected
failures.

> 
>> >> 
>> >> The non-zero slot case has also been discussed in:
>> >> 

[snip]

>> >   
>> 
>> Thank you very much!
>> 
>> I see that the issue has been fixed. The ticket mentions the following.
>> 
>> "What I am observing is that it seems when the slot ID != 0, the guest OS seems
>> to ignore this and we never seem to hit ich9_pm_device_unplug_cb()."
>> 
>> Based on my experience and evaluation, ACPI-based hotplug is more likely to
>> encounter an issue where the guest VM does not respond to an unplug operation.
> 
> I'm not sure it's a good idea to delete device when guest still thinks it's there
> (you can make guesses on QEMU side if it's in use, how useful those are is questionable).

By instrumenting the QEMU functions related to device hotplug and PCI
initialization, we may be able to make informed assumptions.

> 
> as far as I know, ACPI hotplug has no notion of surprise removal (pls educate me if it's not the case),
> so I wouldn't do what you are proposing here at all, it's basically asking for disaster to happen.
> And all this is basically for dealing with abused qemu flexibility.
> 
> Please (re)formulate usecase and make it more clear as what is eludes me
> no matter how many times i've read this cover letter.

Here are some use cases in virtualization and cloud environments:

1. Suppose there is a QEMU user configuration error and a PCI device is attached
to slot 1 of a pcie-root-port that uses ACPI-based hotplug. The device may not
be detected or ejected. As a result, there is no way to detach it from the
user's QEMU instance until the guest VM reboots.

2. For an unknown reason in the customer's guest kernel (Linux, Windows, or
BSD), a PCI device may still be referenced by the guest kernel or its services.
As a result, the guest never writes the eject register for ACPI-based hotplug,
and QEMU cannot detach the device. The customer may blame QEMU for not removing
it. A force-detach option could provide an escape hatch, with a warning that it
may make the VM unstable or insecure.

3. Suppose a VM is stuck because of a guest kernel bug, such as a Linux kernel
panic without a kdump kernel being triggered. The guest kernel is unresponsive.
Force detach could allow the block device to be temporarily attached to another
VM without resetting the currently panicked VM.

4. Provide a mechanism to demonstrate that force detach is unsafe. Otherwise,
users may repeatedly attempt it and mistakenly conclude that QEMU is at fault :)

Thank you very much!

Dongli Zhang

> 
> On positive note:
> 
> What you can try to implement is native PCI-E support for surprise removal.
> How hard that would be I don't know. And I would well expect if one deviates from
> real hw expectations/configs (such as not 0 slot/partial func removal),
> one would quickly stumble upon issues as  that's not what what vendors write/test
> drivers for.
> 
> Even if it's not likely to be used in practice (guest still might not support it),
> it may serve as test-bed for guest drivers.
> 
>> Thank you very much!
>> 
>> Dongli Zhang
>> 
> 

Re: [PATCH 0/8] Force detach PCI devices for ACPI-based and PCIe native hot-unplug

[Dongli Zhang]

On Fri, Sep 4, 2026 5:08:24AM -0700, Igor Mammedov wrote:
> On Fri, 4 Sep 2026 07:22:10 -0400
> "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
>> On Fri, Sep 04, 2026 at 01:07:09PM +0200, Igor Mammedov wrote:
>> > On Thu, 3 Sep 2026 16:17:20 -0400
>> > "Michael S. Tsirkin" <mst@redhat.com> wrote:
>> >   
>> > > On Thu, Sep 03, 2026 at 05:28:35PM +0200, Igor Mammedov wrote:  
>> > > > On Wed, 26 Aug 2026 09:15:47 -0700
>> > > > Dongli Zhang <dongli.zhang@oracle.com> wrote:
>> > > >     
>> > > > > On Mon, Aug 24, 2026 7:44:49AM -0700, Daniel P. Berrangé wrote:    
>> > > > > > On Sun, Aug 23, 2026 at 06:13:30PM -0700, Dongli Zhang wrote:      
>> > > > > >> Hot-unplugging a PCI device can require cooperation from the guest. For
>> > > > > >> ACPI PCI hotplug, QEMU notifies the guest through ACPI and the guest
>> > > > > >> eventually writes the ACPI PCI eject register. For PCIe native hotplug,
>> > > > > >> QEMU notifies the guest through the PCIe hotplug mechanism and waits for
>> > > > > >> the slot unplug flow to complete. Only after that completion does QEMU
>> > > > > >> unrealize the device and emit DEVICE_DELETED.
>> > > > > >> 
>> > > > > >> This can leave a device stuck in the unplug pending state when the guest
>> > > > > >> does not cooperate. Examples include:
>> > > > > >> 
>> > > > > >> 1. The guest has panicked, or the relevant ACPI/PCI hotplug driver is
>> > > > > >> unavailable.
>> > > > > >> 
>> > > > > >> 2. The guest is stalled and cannot handle the hot-unplug event. For
>> > > > > >> example, stalling the Linux [irq/9-acpi] kernel thread can reproduce this
>> > > > > >> for ACPI-based hot-unplug.
>> > > > > >> 
>> > > > > >> 3. The device was attached to a slot that the guest cannot use. For
>> > > > > >> example, a pcie-root-port only supports slot 0. If a device is added to a
>> > > > > >> non-zero slot below a pcie-root-port, the guest may never discover the
>> > > > > >> device and therefore may never complete the unplug request.    
>> > > > 
>> > > > all of above is actually expected, no (functioning) driver => no hotplug/unplug.
>> > > > it's the guest problem. Once device it exposed to guest its life-cycle
>> > > > not longer owned by QEMU.
>> > > > 

[snip]

>> > > > > > 
>> > > > > > If we actually wanted this to remain a warning, then that shutdown
>> > > > > > crash would need to be fixed.
>> > > > > >       
>> > > > > 
>> > > > > Thank you very much!
>> > > > > 
>> > > > > I see that the issue has been fixed. The ticket mentions the following.
>> > > > > 
>> > > > > "What I am observing is that it seems when the slot ID != 0, the guest OS seems
>> > > > > to ignore this and we never seem to hit ich9_pm_device_unplug_cb()."
>> > > > > 
>> > > > > Based on my experience and evaluation, ACPI-based hotplug is more likely to
>> > > > > encounter an issue where the guest VM does not respond to an unplug operation.    
>> > > > 
>> > > > I'm not sure it's a good idea to delete device when guest still thinks it's there
>> > > > (you can make guesses on QEMU side if it's in use, how useful those are is questionable).
>> > > > 
>> > > > as far as I know, ACPI hotplug has no notion of surprise removal (pls educate me if it's not the case),    
>> > > 
>> > > why would it not?
>> > > 
>> > > how do you think you can pull a laptop out of a dock?
>> > > I expect bus check + _STA and config space saying it is gone
>> > > will do exactly that.
>> > > 
>> > > 
>> > > Here's linux code:
>> > > static void acpiphp_check_bridge(struct acpiphp_bridge *bridge)
>> > > {       
>> > >         struct acpiphp_slot *slot;
>> > > 
>> > >         /* Bail out if the bridge is going away. */
>> > >         if (bridge->is_going_away)
>> > >                 return;
>> > > 
>> > >         if (bridge->pci_dev)
>> > >                 pm_runtime_get_sync(&bridge->pci_dev->dev);
>> > >         
>> > >         list_for_each_entry(slot, &bridge->slots, node) {
>> > >                 struct pci_bus *bus = slot->bus;
>> > >                 struct pci_dev *dev, *tmp;
>> > > 
>> > >                 if (slot_no_hotplug(slot)) {
>> > >                         ; /* do nothing */
>> > >                 } else if (device_status_valid(get_slot_status(slot))) {
>> > >                         /* remove stale devices if any */
>> > >                         list_for_each_entry_safe_reverse(dev, tmp,
>> > >                                                          &bus->devices, bus_list)
>> > >                                 if (PCI_SLOT(dev->devfn) == slot->device)
>> > >                                         trim_stale_devices(dev);
>> > > 
>> > >                         /* configure all functions */
>> > >                         enable_slot(slot, true);
>> > >                 } else {
>> > >                         disable_slot(slot);
>> > >                 }
>> > >         }
>> > > 
>> > >         if (bridge->pci_dev)
>> > >                 pm_runtime_put(&bridge->pci_dev->dev);
>> > > }
>> > > 
>> > > 
>> > > so weirdly it wants bus check on a parent bus, otherwise it will
>> > > not trim devices?  probably a bug, but easy to work around.  
>> > 
>> > Modern docks would use native pcie surprise removal path.
>> > 
>> > As for ACPI, my old laptop, had an unlock button => _LCK
>> > and that relied on OS processing ACPI events, not so surprise.
>> > 
>> > There might have been ACPI/hybrid docks that did surprise removal,
>> > but then one need to find one and model after that instead of 
>> > just blanket force removal. (likely out come would a doc device
>> > support only, not an arbitrary device removal)
>> > 
>> > (not the case described in this series, though. hence my request to clarify usecase)
>> > 
>> > from what I see in spec there is _RMV method that says that device
>> > supports surprise removal that can be used for devices that support it.
>> > However I would hesitate very much to blank apply it to every PCI device.
>> > (it's not even realistic to ask for proving safe tear down across various
>> > drivers and OSes/versions)
>> > 
>> > Rather than a knee jerk treatment of misconfig consequences,
>> > I'd rather see patches to prevent misconfig in the 1st place
>> > (subj to deprecation but doable).
>> > 
>> > As for the cases where OS mis-behaves (apcihp thread starvation,...),
>> > fixing guest to follow hotplug contract is a proper place to do it.
>> > 
>> > On QEMU side we have it covered as well. If unplug was not processed,
>> > mgmt is free to repeat action.  
>> 
>> 
>> Sorry if I am unclear. I just meant that it looks like we
>> can support surprise removal with ACPI just by reporting
>> bus check events on the parent.
> 
> maybe, but that ain't SPECed and might be OS specific.
> 
> The way I've read the cover letter, that won't work for mentioned mis-config cases.
> Also what would happen on bus-check 'cleanup' would be a lottery.
> hence I'm for being safe here.
> 
> It's better to implement native PCIE surprise removal if that's really needed.
> 

Suppose many x86 users use q35 and pcie-root-port. Since commit 17858a169508
("hw/acpi/ich9: Set ACPI PCI hot-plug as default on Q35"), ACPI-based hotplug
has been the default for q35. arm64 still uses native PCIe hotplug.

Therefore, in my opinion, it is more crucial to support ACPI-based hotplug than
native PCIe hotplug. In addition, native PCIe hotplug can still detach a PCI
device even when the device is erroneously attached to slot 1 of a pcie-root-port.

Although surprise removal is not explicitly specified and may be OS-specific, my
understanding is that it involves two steps:

1. Force-detach the PCI device.
2. Use a mechanism to notify the guest VM that the device is no longer present.

Therefore, may I assume that this can address the use cases mentioned in the
cover letter?

Thank you very much!

Dongli Zhang

Re: [PATCH 0/8] Force detach PCI devices for ACPI-based and PCIe native hot-unplug

[Michael S. Tsirkin]

On Mon, Sep 07, 2026 at 01:26:57AM -0700, Dongli Zhang wrote:
> 
> 
> On Thu, Sep 3, 2026 8:28:35AM -0700, Igor Mammedov wrote:
> > On Wed, 26 Aug 2026 09:15:47 -0700
> > Dongli Zhang <dongli.zhang@oracle.com> wrote:
> > 
> >> On Mon, Aug 24, 2026 7:44:49AM -0700, Daniel P. Berrangé wrote:
> >> > On Sun, Aug 23, 2026 at 06:13:30PM -0700, Dongli Zhang wrote:  
> >> >> Hot-unplugging a PCI device can require cooperation from the guest. For
> >> >> ACPI PCI hotplug, QEMU notifies the guest through ACPI and the guest
> >> >> eventually writes the ACPI PCI eject register. For PCIe native hotplug,
> >> >> QEMU notifies the guest through the PCIe hotplug mechanism and waits for
> >> >> the slot unplug flow to complete. Only after that completion does QEMU
> >> >> unrealize the device and emit DEVICE_DELETED.
> >> >> 
> >> >> This can leave a device stuck in the unplug pending state when the guest
> >> >> does not cooperate. Examples include:
> >> >> 
> >> >> 1. The guest has panicked, or the relevant ACPI/PCI hotplug driver is
> >> >> unavailable.
> >> >> 
> >> >> 2. The guest is stalled and cannot handle the hot-unplug event. For
> >> >> example, stalling the Linux [irq/9-acpi] kernel thread can reproduce this
> >> >> for ACPI-based hot-unplug.
> >> >> 
> >> >> 3. The device was attached to a slot that the guest cannot use. For
> >> >> example, a pcie-root-port only supports slot 0. If a device is added to a
> >> >> non-zero slot below a pcie-root-port, the guest may never discover the
> >> >> device and therefore may never complete the unplug request.
> > 
> > all of above is actually expected, no (functioning) driver => no hotplug/unplug.
> > it's the guest problem. Once device it exposed to guest its life-cycle
> > not longer owned by QEMU.
> > 
> > That's what one would see in real hw as well, you press eject button
> > but it will not do anything if OS doesn't process it.
> > also see comment at the end.
> 
> Users are generally more tolerant of issues with real hardware.
> 
> In virtualization and cloud environments, PCI hotplug is more commonly used for
> NICs and storage devices. Users are less tolerant of disruption or unexpected
> failures.


Simply put, surprise removal exists in the hardware.  Emulating that
makes sense, at a high level. Nor is it too hard.
However, guests, especially Linux, do not
handle it all that well generally. Exactly because users
would tend to impatiently reach for that tool, then
blame QEMU after a crash, we avoided emulating that.


I'd expect much more in the way of research into how guests behave,
perhaps some ways to limit it to devices that work well, and
likely some linux patches to make it work better, before we commit to
supporting such interfaces.


> > 
> >> >> 
> >> >> The non-zero slot case has also been discussed in:
> >> >> 
> 
> [snip]
> 
> >> >   
> >> 
> >> Thank you very much!
> >> 
> >> I see that the issue has been fixed. The ticket mentions the following.
> >> 
> >> "What I am observing is that it seems when the slot ID != 0, the guest OS seems
> >> to ignore this and we never seem to hit ich9_pm_device_unplug_cb()."
> >> 
> >> Based on my experience and evaluation, ACPI-based hotplug is more likely to
> >> encounter an issue where the guest VM does not respond to an unplug operation.
> > 
> > I'm not sure it's a good idea to delete device when guest still thinks it's there
> > (you can make guesses on QEMU side if it's in use, how useful those are is questionable).
> 
> By instrumenting the QEMU functions related to device hotplug and PCI
> initialization, we may be able to make informed assumptions.

So try. But just know that pci initialization is commonly done by firmware, not
the driver.

> > 
> > as far as I know, ACPI hotplug has no notion of surprise removal (pls educate me if it's not the case),
> > so I wouldn't do what you are proposing here at all, it's basically asking for disaster to happen.
> > And all this is basically for dealing with abused qemu flexibility.
> > 
> > Please (re)formulate usecase and make it more clear as what is eludes me
> > no matter how many times i've read this cover letter.
> 
> Here are some use cases in virtualization and cloud environments:
> 
> 1. Suppose there is a QEMU user configuration error and a PCI device is attached
> to slot 1 of a pcie-root-port that uses ACPI-based hotplug. The device may not
> be detected or ejected. As a result, there is no way to detach it from the
> user's QEMU instance until the guest VM reboots.

sounds vague. if users can not configure qemu what are the chances
they will use force detach responsibly?

> 2. For an unknown reason in the customer's guest kernel (Linux, Windows, or
> BSD), a PCI device may still be referenced by the guest kernel or its services.
> As a result, the guest never writes the eject register for ACPI-based hotplug,
> and QEMU cannot detach the device. The customer may blame QEMU for not removing
> it. A force-detach option could provide an escape hatch, with a warning that it
> may make the VM unstable or insecure.

So just reboot the guest.

> 3. Suppose a VM is stuck because of a guest kernel bug, such as a Linux kernel
> panic without a kdump kernel being triggered. The guest kernel is unresponsive.
> Force detach could allow the block device to be temporarily attached to another
> VM without resetting the currently panicked VM.

So just reboot the guest.

> 4. Provide a mechanism to demonstrate that force detach is unsafe. Otherwise,
> users may repeatedly attempt it and mistakenly conclude that QEMU is at fault :)

We have that - we don't support unsafe detach.

> Thank you very much!
> 
> Dongli Zhang
> 
> > 
> > On positive note:
> > 
> > What you can try to implement is native PCI-E support for surprise removal.
> > How hard that would be I don't know. And I would well expect if one deviates from
> > real hw expectations/configs (such as not 0 slot/partial func removal),
> > one would quickly stumble upon issues as  that's not what what vendors write/test
> > drivers for.
> > 
> > Even if it's not likely to be used in practice (guest still might not support it),
> > it may serve as test-bed for guest drivers.
> > 
> >> Thank you very much!
> >> 
> >> Dongli Zhang
> >> 
> > 

Re: [PATCH 0/8] Force detach PCI devices for ACPI-based and PCIe native hot-unplug

[Dongli Zhang]

On Mon, Sep 7, 2026 1:55:42AM -0700, Michael S. Tsirkin wrote:
> On Mon, Sep 07, 2026 at 01:26:57AM -0700, Dongli Zhang wrote:
>> 
>> 
>> On Thu, Sep 3, 2026 8:28:35AM -0700, Igor Mammedov wrote:
>> > On Wed, 26 Aug 2026 09:15:47 -0700
>> > Dongli Zhang <dongli.zhang@oracle.com> wrote:
>> > 
>> >> On Mon, Aug 24, 2026 7:44:49AM -0700, Daniel P. Berrangé wrote:
>> >> > On Sun, Aug 23, 2026 at 06:13:30PM -0700, Dongli Zhang wrote:  
>> >> >> Hot-unplugging a PCI device can require cooperation from the guest. For
>> >> >> ACPI PCI hotplug, QEMU notifies the guest through ACPI and the guest
>> >> >> eventually writes the ACPI PCI eject register. For PCIe native hotplug,
>> >> >> QEMU notifies the guest through the PCIe hotplug mechanism and waits for
>> >> >> the slot unplug flow to complete. Only after that completion does QEMU
>> >> >> unrealize the device and emit DEVICE_DELETED.
>> >> >> 
>> >> >> This can leave a device stuck in the unplug pending state when the guest
>> >> >> does not cooperate. Examples include:
>> >> >> 
>> >> >> 1. The guest has panicked, or the relevant ACPI/PCI hotplug driver is
>> >> >> unavailable.
>> >> >> 
>> >> >> 2. The guest is stalled and cannot handle the hot-unplug event. For
>> >> >> example, stalling the Linux [irq/9-acpi] kernel thread can reproduce this
>> >> >> for ACPI-based hot-unplug.
>> >> >> 
>> >> >> 3. The device was attached to a slot that the guest cannot use. For
>> >> >> example, a pcie-root-port only supports slot 0. If a device is added to a
>> >> >> non-zero slot below a pcie-root-port, the guest may never discover the
>> >> >> device and therefore may never complete the unplug request.
>> > 
>> > all of above is actually expected, no (functioning) driver => no hotplug/unplug.
>> > it's the guest problem. Once device it exposed to guest its life-cycle
>> > not longer owned by QEMU.
>> > 
>> > That's what one would see in real hw as well, you press eject button
>> > but it will not do anything if OS doesn't process it.
>> > also see comment at the end.
>> 
>> Users are generally more tolerant of issues with real hardware.
>> 
>> In virtualization and cloud environments, PCI hotplug is more commonly used for
>> NICs and storage devices. Users are less tolerant of disruption or unexpected
>> failures.
> 
> 
> Simply put, surprise removal exists in the hardware.  Emulating that
> makes sense, at a high level. Nor is it too hard.
> However, guests, especially Linux, do not
> handle it all that well generally. Exactly because users
> would tend to impatiently reach for that tool, then
> blame QEMU after a crash, we avoided emulating that.
> 
> 
> I'd expect much more in the way of research into how guests behave,
> perhaps some ways to limit it to devices that work well, and
> likely some linux patches to make it work better, before we commit to
> supporting such interfaces.

Here is my summary of the current situation and plan:

1. Folks are not against adding an option to force-detach a PCI device. It
should emulate surprise removal, which exists in real hardware. This requires
adding the emulation to QEMU.

2. However, the more important issue is ensuring that surprise removal works
well in guest operating systems, such as Linux and Windows, and for a defined
subset of devices, including virtio devices, VFIO-assigned devices, Intel,
Mellanox, and Broadcom NICs, and NVMe devices.

From QEMU’s perspective, it is preferable not to expose an interface that may
cause a guest crash and lead users to blame QEMU for a guest-side limitation.

3. More research and development work is required to support surprise removal in
both QEMU and guest operating systems, especially the Linux kernel.

4. So far, from QEMU's perspective, everyone suggest rebooting the guest VM for
many of the use cases I mentioned

5. By instrumenting QEMU code and possibly adding additional metadata, we can
track whether a PCI device has been accessed by the guest VM, especially during
PCI hotplug. If the device has never been used by the guest VM, it may be safer
to force-detach it.

> 
> 
>> > 
>> >> >> 
>> >> >> The non-zero slot case has also been discussed in:
>> >> >> 
>> 
>> [snip]
>> 
>> >> >   
>> >> 
>> >> Thank you very much!
>> >> 
>> >> I see that the issue has been fixed. The ticket mentions the following.
>> >> 
>> >> "What I am observing is that it seems when the slot ID != 0, the guest OS seems
>> >> to ignore this and we never seem to hit ich9_pm_device_unplug_cb()."
>> >> 
>> >> Based on my experience and evaluation, ACPI-based hotplug is more likely to
>> >> encounter an issue where the guest VM does not respond to an unplug operation.
>> > 
>> > I'm not sure it's a good idea to delete device when guest still thinks it's there
>> > (you can make guesses on QEMU side if it's in use, how useful those are is questionable).
>> 
>> By instrumenting the QEMU functions related to device hotplug and PCI
>> initialization, we may be able to make informed assumptions.
> 
> So try. But just know that pci initialization is commonly done by firmware, not
> the driver.

Thank you very much for the confirmation. This also provides users with
telemetry to narrow down the potential reasons why QEMU does not send the
DEVICE_DELETED QMP event.

> 
>> > 
>> > as far as I know, ACPI hotplug has no notion of surprise removal (pls educate me if it's not the case),
>> > so I wouldn't do what you are proposing here at all, it's basically asking for disaster to happen.
>> > And all this is basically for dealing with abused qemu flexibility.
>> > 
>> > Please (re)formulate usecase and make it more clear as what is eludes me
>> > no matter how many times i've read this cover letter.
>> 
>> Here are some use cases in virtualization and cloud environments:
>> 
>> 1. Suppose there is a QEMU user configuration error and a PCI device is attached
>> to slot 1 of a pcie-root-port that uses ACPI-based hotplug. The device may not
>> be detected or ejected. As a result, there is no way to detach it from the
>> user's QEMU instance until the guest VM reboots.
> 
> sounds vague. if users can not configure qemu what are the chances
> they will use force detach responsibly?

I should have clarified the difference between the QEMU user and the VM owner.
The QEMU user is the software that manages VMs, such as libvirt or any other
software that communicates with QEMU through QMP. The VM owner does not have
access to QEMU.

However, any mistake made by the QEMU user can affect the VM owner. For example,
if the QEMU user mistakenly adds a PCI device to slot 1 of a pcie-root-port, the
VM owner cannot detach the device from the QEMU instance until the guest reboots.

In this case, the QEMU user is at fault, but the VM owner is affected.

A force or surprise removal option helps the QEMU user recover from a
configuration error without requiring action from the VM owner.

> 
>> 2. For an unknown reason in the customer's guest kernel (Linux, Windows, or
>> BSD), a PCI device may still be referenced by the guest kernel or its services.
>> As a result, the guest never writes the eject register for ACPI-based hotplug,
>> and QEMU cannot detach the device. The customer may blame QEMU for not removing
>> it. A force-detach option could provide an escape hatch, with a warning that it
>> may make the VM unstable or insecure.
> 
> So just reboot the guest.

I agree.

Sometimes, the VM owner blames QEMU for not detaching a PCI device, even though
QEMU is technically waiting for the guest VM to write to the EJ register :(

> 
>> 3. Suppose a VM is stuck because of a guest kernel bug, such as a Linux kernel
>> panic without a kdump kernel being triggered. The guest kernel is unresponsive.
>> Force detach could allow the block device to be temporarily attached to another
>> VM without resetting the currently panicked VM.
> 
> So just reboot the guest.

I agree.

Sometimes, the VM owner blames QEMU for not detaching a PCI device, even though
QEMU is technically waiting for the guest VM to write to the EJ register :(

> 
>> 4. Provide a mechanism to demonstrate that force detach is unsafe. Otherwise,
>> users may repeatedly attempt it and mistakenly conclude that QEMU is at fault :)
> 
> We have that - we don't support unsafe detach.

Thank you very much!

Dongli Zhang

Re: [PATCH 0/8] Force detach PCI devices for ACPI-based and PCIe native hot-unplug

[Michael S. Tsirkin]

On Mon, Sep 07, 2026 at 02:06:29PM -0700, Dongli Zhang wrote:
> 5. By instrumenting QEMU code and possibly adding additional metadata, we can
> track whether a PCI device has been accessed by the guest VM, especially during
> PCI hotplug. If the device has never been used by the guest VM, it may be safer
> to force-detach it.

this last one I am not sure about. as I said, devices are commonly
accessed by firmware.