[RFC 0/1] Forward AER errors to guest

[RFC 0/1] Forward AER errors to guest

[Satyanarayana K V P]

Today, vfio-pci unconditionally stops the VM when any error event is
reported by device. This prevents guest-driven error handling and recovery
for platforms that support PCIe AER.

This series adds an optional vfio-pci extension parameter,
"x-forward-aer=on", to forward AER errors to the guest instead of
forcing an immediate VM stop. If the endpoint supports AER, the error is
forwarded directly; otherwise, checks the upstream PCIe bridge and
forwards the error there when supported.If neither device nor PCIe bridge
supports AER, the VM is immediately stopped when the error is reported.

Requesting for comments from vfio maintainers and AER experts for this
patch series.

Signed-off-by: Satyanarayana K V P <satyanarayana.k.v.p@intel.com>
Cc: Michał Winiarski <michal.winiarski@intel.com>
Cc: Michal Wajdeczko <michal.wajdeczko@intel.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Alex Williamson <alex@shazbot.org>
Cc: Cédric Le Goater <clg@redhat.com>

Satyanarayana K V P (1):
  vfio/pci: Forward AER errors to guest optionally

 hw/vfio/pci.c | 80 ++++++++++++++++++++++++++++++++++++++++++++-------
 hw/vfio/pci.h |  1 +
 2 files changed, 70 insertions(+), 11 deletions(-)

-- 
2.43.0

[RFC 1/1] vfio/pci: Forward AER errors to guest optionally

[Satyanarayana K V P]

Currently QEMU unconditionally stops VM upon receiving any unrecoverable
error from the device. This prevents recovery of the device within the
guest.

Upon receiving the AER error from device, an error is injected to guest if
the device supports AER. If the device does not support AER, check whether
the upstream PCIe bridge supports it and forward the error to the bridge.

The error forward is gated with an optional extension parameter and can be
enabled with

-device vfio-pci,host=<BDF>,...,x-forward-aer=on

Signed-off-by: Satyanarayana K V P <satyanarayana.k.v.p@intel.com>
Cc: Michał Winiarski <michal.winiarski@intel.com>
Cc: Michal Wajdeczko <michal.wajdeczko@intel.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Alex Williamson <alex@shazbot.org>
Cc: Cédric Le Goater <clg@redhat.com>
---
 hw/vfio/pci.c | 80 ++++++++++++++++++++++++++++++++++++++++++++-------
 hw/vfio/pci.h |  1 +
 2 files changed, 70 insertions(+), 11 deletions(-)

diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
index 9c06b25e63..1913d18ff5 100644
--- a/hw/vfio/pci.c
+++ b/hw/vfio/pci.c
@@ -2628,6 +2628,14 @@ static void vfio_add_ext_cap(VFIOPCIDevice *vdev)
                 pcie_add_capability(pdev, cap_id, cap_ver, next, size);
             }
             break;
+        case PCI_EXT_CAP_ID_ERR:
+            if (pcie_aer_init(pdev, cap_ver, next, size, &err) < 0) {
+                warn_report_err(err);
+                err = NULL;
+                /* Mark capability as absent on failure to initialize */
+                pdev->exp.aer_cap = 0;
+            }
+            break;
         /*
          * VFIO kernel does not expose the PASID CAP today. We may synthesize
          * one later through IOMMUFD APIs. If VFIO ever starts exposing it,
@@ -3124,26 +3132,68 @@ void vfio_pci_put_device(VFIOPCIDevice *vdev)
     g_free(vdev->msix);
 }
 
-static void vfio_err_notifier_handler(void *opaque)
+static void vfio_err_notify_guest(VFIOPCIDevice *vdev)
 {
-    VFIOPCIDevice *vdev = opaque;
+    PCIDevice *pdev = PCI_DEVICE(vdev);
+    PCIDevice *pbridge_dev;
+    PCIEAERErr aer_err = {
+        .status = PCI_ERR_UNC_MALF_TLP,
+        .flags = 0,
+    };
+    int ret;
 
-    if (!event_notifier_test_and_clear(&vdev->err_notifier)) {
-        return;
+    if (!vdev->forward_aer) {
+        error_report("%s(%s)"
+                     "Unrecoverable error detected and could not notify guest. "
+                     "Please collect any data possible and then kill the guest",
+                     __func__, vdev->vbasedev.name);
+        goto vm_stop_out;
     }
 
     /*
-     * TBD. Retrieve the error details and decide what action
-     * needs to be taken. One of the actions could be to pass
-     * the error to the guest and have the guest driver recover
-     * from the error. This requires that PCIe capabilities be
-     * exposed to the guest. For now, we just terminate the
-     * guest to contain the error.
+     * If the device does not have AER capability and still configured to
+     * forward error to guest, try to find a PCIe bridge with AER capability and
+     * forward the error to it. If neither the device nor the PCIe bridge supports
+     * AER, the VM is immediately stopped when the error is reported.
      */
+    if (!pdev->exp.aer_cap) {
+        pbridge_dev = pci_bridge_get_device(pci_get_bus(pdev));
+        if (pbridge_dev && pbridge_dev->exp.aer_cap) {
+            warn_report("Forwarding error to PCIe bridge due to lack of AER capability on device");
+            pdev = pbridge_dev;
+        } else {
+            error_report("%s(%s) No AER capable device is found, stopping VM",
+                         __func__, vdev->vbasedev.name);
+            goto vm_stop_out;
+        }
+    }
+
+    aer_err.source_id = pci_requester_id(pdev);
+
+    ret = pcie_aer_inject_error(pdev, &aer_err);
+    if (ret < 0) {
+        error_report("%s(%s): AER injection failed (%d)", __func__, vdev->vbasedev.name, ret);
+        goto vm_stop_out;
+    }
 
-    error_report("%s(%s) Unrecoverable error detected. Please collect any data possible and then kill the guest", __func__, vdev->vbasedev.name);
+    return;
 
+vm_stop_out:
     vm_stop(RUN_STATE_INTERNAL_ERROR);
+    return;
+}
+
+static void vfio_err_notifier_handler(void *opaque)
+{
+    VFIOPCIDevice *vdev = opaque;
+
+    if (!event_notifier_test_and_clear(&vdev->err_notifier)) {
+        return;
+    }
+
+    error_report("%s(%s) Unrecoverable error detected for the device",
+                 __func__, vdev->vbasedev.name);
+    vfio_err_notify_guest(vdev);
 }
 
 /*
@@ -3624,6 +3674,9 @@ static void vfio_exitfn(PCIDevice *pdev)
     VFIOPCIDevice *vdev = VFIO_PCI_DEVICE(pdev);
     VFIODevice *vbasedev = &vdev->vbasedev;
 
+    if (pdev->exp.aer_cap) {
+        pcie_aer_exit(pdev);
+    }
     vfio_unregister_req_notifier(vdev);
     vfio_unregister_err_notifier(vdev);
     pci_device_set_intx_routing_notifier(pdev, NULL);
@@ -3815,6 +3868,7 @@ static const Property vfio_pci_properties[] = {
     DEFINE_PROP_BOOL("skip-vsc-check", VFIOPCIDevice, skip_vsc_check, true),
     DEFINE_PROP_UINT16("x-vpasid-cap-offset", VFIOPCIDevice,
                        vpasid_cap_offset, 0),
+    DEFINE_PROP_BOOL("x-forward-aer", VFIOPCIDevice, forward_aer, false),
 };
 
 static void vfio_pci_set_fd(Object *obj, const char *str, Error **errp)
@@ -3973,6 +4027,10 @@ static void vfio_pci_class_init(ObjectClass *klass, const void *data)
                                           "a vIOMMU. A value of 0 (default) places the capability at the "
                                           "end of the extended configuration space. The offset must be "
                                           "4-byte aligned and within the PCIe extended configuration space");
+    object_class_property_set_description(klass,
+                                          "x-forward-aer",
+                                          "Forward Advanced Error Reporting (AER) events to the guest "
+                                          "with generic error status");
 }
 
 static const TypeInfo vfio_pci_info = {
diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h
index c3a1f53d35..b1d19c4c9e 100644
--- a/hw/vfio/pci.h
+++ b/hw/vfio/pci.h
@@ -187,6 +187,7 @@ struct VFIOPCIDevice {
     bool defer_kvm_irq_routing;
     bool clear_parent_atomics_on_exit;
     bool skip_vsc_check;
+    bool forward_aer;
     uint16_t vpasid_cap_offset;
     VFIODisplay *dpy;
     Notifier irqchip_change_notifier;
-- 
2.43.0

Re: [RFC 0/1] Forward AER errors to guest

[Alex Williamson]

On Fri, Jul 3, 2026, at 5:13 AM, Satyanarayana K V P wrote:
> Today, vfio-pci unconditionally stops the VM when any error event is
> reported by device. This prevents guest-driven error handling and recovery
> for platforms that support PCIe AER.
>
> This series adds an optional vfio-pci extension parameter,
> "x-forward-aer=on", to forward AER errors to the guest instead of
> forcing an immediate VM stop. If the endpoint supports AER, the error is
> forwarded directly; otherwise, checks the upstream PCIe bridge and
> forwards the error there when supported.If neither device nor PCIe bridge
> supports AER, the VM is immediately stopped when the error is reported.

The error eventfd is signaled for err_detected in the host, this is the beginning of the host error handling and the point at which drivers should stop accessing the device until the resume callback.  Letting the VMM continue at this point does the opposite of that.  Thanks,

Alex

Re: [RFC 0/1] Forward AER errors to guest

[Michał Winiarski]

On Fri, Jul 03, 2026 at 07:59:46AM -0600, Alex Williamson wrote:
> 
> 
> On Fri, Jul 3, 2026, at 5:13 AM, Satyanarayana K V P wrote:
> > Today, vfio-pci unconditionally stops the VM when any error event is
> > reported by device. This prevents guest-driven error handling and recovery
> > for platforms that support PCIe AER.
> >
> > This series adds an optional vfio-pci extension parameter,
> > "x-forward-aer=on", to forward AER errors to the guest instead of
> > forcing an immediate VM stop. If the endpoint supports AER, the error is
> > forwarded directly; otherwise, checks the upstream PCIe bridge and
> > forwards the error there when supported.If neither device nor PCIe bridge
> > supports AER, the VM is immediately stopped when the error is reported.
> 
> The error eventfd is signaled for err_detected in the host, this is the beginning of the host error handling and the point at which drivers should stop accessing the device until the resume callback.  Letting the VMM continue at this point does the opposite of that.  Thanks,
> 
> Alex

Hi,

Today - the default error handler used by VFIO is not implementing the
.resume() callback and just signals err_trigger and returns
PCI_ERS_RESULT_CAN_RECOVER as part of .error_detected().
Qemu registers a callback to be called upon err_trigger signal (using
one of the poll/select syscall variants), the callback calls
vm_stop(RUN_STATE_INTERNAL_ERROR) - this is a runstate that can't
continue, the VM is effectively killed (qemu needs to trigger VM reset
to recover).

The AER forward was proposed as a relatively simple opt-in mechanism to
avoid hitting RUN_STATE_INTERNAL_ERROR and be able to handle errors for
SR-IOV VFs using VFIO. For uncorrectable errors with SR-IOV, PF driver
can be the entity that handles the device recovery. The VF driver needs
to stop using the device until it is recovered by PF, and reinitialize
any state that was lost as part of the recovery action.
Forwarding AER would allow the VM to move forward without introducing
any new uAPI.

If we would want to build a generic error recovery mechanism, we would
need to duplicate more of the err_handler_t callbacks as VFIO uAPI.

If the recovery is handled by other entity running on the host (VFIO
variant? Or for SR-IOV, the PF driver), we could also go into a
different runstate (RUN_STATE_PAUSED? something that doesn't require
full VM reset) and add a VFIO uAPI that can propagate the .resume() into
userspace.

If we would want the SW running inside the VM to handle the recovery...
it becomes more involved, and it requires a more substantial solution
that wasn't meant to be addressed here.

So - we're letting the VMM continue but we're also informing it that
something went wrong with the device, so that the SW running inside the
VM can do its own action.
If it's a virtual function, the driver can potentially fully recover
after the host does its thing.
If it's a regular native function, it will probably need to go into
PCI_ERS_RESULT_DISCONNECT, which, for most usecases is still a better
scenario then having to reset the entire VM.

If we would want to absolutely make sure that the users are not able to
issue any IO across error recovery, we can probably unmap (or zero-map)
the BARs at VFIO driver level, similar to what's done during reset.

Thanks,
-Michał

Re: [RFC 0/1] Forward AER errors to guest

[Alex Williamson]

On Tue, 7 Jul 2026 23:05:33 +0200
Michał Winiarski <michal.winiarski@intel.com> wrote:

> On Fri, Jul 03, 2026 at 07:59:46AM -0600, Alex Williamson wrote:
> > 
> > 
> > On Fri, Jul 3, 2026, at 5:13 AM, Satyanarayana K V P wrote:  
> > > Today, vfio-pci unconditionally stops the VM when any error event is
> > > reported by device. This prevents guest-driven error handling and recovery
> > > for platforms that support PCIe AER.
> > >
> > > This series adds an optional vfio-pci extension parameter,
> > > "x-forward-aer=on", to forward AER errors to the guest instead of
> > > forcing an immediate VM stop. If the endpoint supports AER, the error is
> > > forwarded directly; otherwise, checks the upstream PCIe bridge and
> > > forwards the error there when supported.If neither device nor PCIe bridge
> > > supports AER, the VM is immediately stopped when the error is reported.  
> > 
> > The error eventfd is signaled for err_detected in the host, this is
> > the beginning of the host error handling and the point at which
> > drivers should stop accessing the device until the resume callback.
> >  Letting the VMM continue at this point does the opposite of that.
> > Thanks,
> > 
> > Alex  
> 
> Hi,
> 
> Today - the default error handler used by VFIO is not implementing the
> .resume() callback and just signals err_trigger and returns
> PCI_ERS_RESULT_CAN_RECOVER as part of .error_detected().
> Qemu registers a callback to be called upon err_trigger signal (using
> one of the poll/select syscall variants), the callback calls
> vm_stop(RUN_STATE_INTERNAL_ERROR) - this is a runstate that can't
> continue, the VM is effectively killed (qemu needs to trigger VM reset
> to recover).

Yes, this is the extent of the implementation currently.
 
> The AER forward was proposed as a relatively simple opt-in mechanism
> to avoid hitting RUN_STATE_INTERNAL_ERROR and be able to handle
> errors for SR-IOV VFs using VFIO. For uncorrectable errors with
> SR-IOV, PF driver can be the entity that handles the device recovery.
> The VF driver needs to stop using the device until it is recovered by
> PF, and reinitialize any state that was lost as part of the recovery
> action. Forwarding AER would allow the VM to move forward without
> introducing any new uAPI.

Interesting, nothing in the proposal suggested this was explicitly for
SR-IOV VFs.
 
> If we would want to build a generic error recovery mechanism, we would
> need to duplicate more of the err_handler_t callbacks as VFIO uAPI.
> 
> If the recovery is handled by other entity running on the host (VFIO
> variant? Or for SR-IOV, the PF driver), we could also go into a
> different runstate (RUN_STATE_PAUSED? something that doesn't require
> full VM reset) and add a VFIO uAPI that can propagate the .resume()
> into userspace.

And a .slot_reset() to actually restore state if PF is reset (we
currently rely on that happening on .release()), and we'll also need to
teardown things like interrupts (also currently reliant on release).

> If we would want the SW running inside the VM to handle the
> recovery... it becomes more involved, and it requires a more
> substantial solution that wasn't meant to be addressed here.

Note that the current error eventfd is triggered on any uncorrected
error, fatal or non-fatal.  How does the VM know which it is?  There
are certainly hacks we can do to limp along a little further with the
current limited implementation, but nothing that seems to actually get
us closer to a general purpose, supportable feature.

> So - we're letting the VMM continue but we're also informing it that
> something went wrong with the device, so that the SW running inside
> the VM can do its own action.
> If it's a virtual function, the driver can potentially fully recover
> after the host does its thing.
> If it's a regular native function, it will probably need to go into
> PCI_ERS_RESULT_DISCONNECT, which, for most usecases is still a better
> scenario then having to reset the entire VM.

We've been discussing this internally as well.  The VM can perform its
own recovery, resetting and re-initializing devices from a VM
perspective, but it can't "handle" the host recovery process.  In fact,
I think at best it could pause and observe the host process, resuming
and injecting an error after host recovery completes.

> If we would want to absolutely make sure that the users are not able
> to issue any IO across error recovery, we can probably unmap (or
> zero-map) the BARs at VFIO driver level, similar to what's done
> during reset.

Yes, the shape I imagined would immediately block all access to the
device on .err_detected(), zap BARs, fail access, and signal the
existing error eventfd.  The VMM would need to handle the potential
sigbus it might see before noticing the eventfd, determine whether it
maps to the mmio space of a device and call a new device feature ioctl
to test whether the device is in an error state (hand waving how QEMU
gets from the trap handler to a point where the VM is paused and we can
poll an ioctl).  Flag bits on the GET of that feature ioctl might
indicate Fatal/Non-Fatal, InProgress, DeviceReset, and Failed.  Perhaps
even a sequence number and/or SET on the feature ioctl might implement
a W1C style acknowledgment - some mechanism by which the user can track
whether they've missed an event.  QEMU could autonomously perform a
surprise hot-unplug on Fatal or Failed, or otherwise pause the VM for
InProgress to clear and inject an appropriate error when it does.
APEI/GHES handling in the guest might be particularly useful to
indicate to the VM whether a device reset has already been performed.
In fact, the whole process doesn't seem too dissimilar to firmware
first error handling that runs underneath the host OS on bare metal.

If you have better ideas how to handle it properly, please share and
correct.  I don't however see much use in nudging out a slightly
improved use case with a narrowly focused, experimental flag that would
never get picked up by tools like libvirt.  Thanks,

Alex

RE: [RFC 0/1] Forward AER errors to guest

[Shameer Kolothum Thodi]

> -----Original Message-----
> From: Alex Williamson <alex.williamson@nvidia.com>
> Sent: 07 July 2026 23:13
> To: Michał Winiarski <michal.winiarski@intel.com>; Shameer Kolothum Thodi
> <skolothumtho@nvidia.com>
> Cc: Satyanarayana K V P <satyanarayana.k.v.p@intel.com>; qemu-
> devel@nongnu.org; Michal Wajdeczko <michal.wajdeczko@intel.com>;
> Matthew Brost <matthew.brost@intel.com>; Cédric Le Goater
> <clg@redhat.com>
> Subject: Re: [RFC 0/1] Forward AER errors to guest
> 
> On Tue, 7 Jul 2026 23:05:33 +0200
> Michał Winiarski <michal.winiarski@intel.com> wrote:
> 
> > On Fri, Jul 03, 2026 at 07:59:46AM -0600, Alex Williamson wrote:
> > >
> > >
> > > On Fri, Jul 3, 2026, at 5:13 AM, Satyanarayana K V P wrote:
> > > > Today, vfio-pci unconditionally stops the VM when any error event is
> > > > reported by device. This prevents guest-driven error handling and
> recovery
> > > > for platforms that support PCIe AER.
> > > >
> > > > This series adds an optional vfio-pci extension parameter,
> > > > "x-forward-aer=on", to forward AER errors to the guest instead of
> > > > forcing an immediate VM stop. If the endpoint supports AER, the error is
> > > > forwarded directly; otherwise, checks the upstream PCIe bridge and
> > > > forwards the error there when supported.If neither device nor PCIe
> bridge
> > > > supports AER, the VM is immediately stopped when the error is
> reported.
> > >
> > > The error eventfd is signaled for err_detected in the host, this is
> > > the beginning of the host error handling and the point at which
> > > drivers should stop accessing the device until the resume callback.
> > >  Letting the VMM continue at this point does the opposite of that.
> > > Thanks,
> > >
> > > Alex
> >
> > Hi,
> >
> > Today - the default error handler used by VFIO is not implementing the
> > .resume() callback and just signals err_trigger and returns
> > PCI_ERS_RESULT_CAN_RECOVER as part of .error_detected().
> > Qemu registers a callback to be called upon err_trigger signal (using
> > one of the poll/select syscall variants), the callback calls
> > vm_stop(RUN_STATE_INTERNAL_ERROR) - this is a runstate that can't
> > continue, the VM is effectively killed (qemu needs to trigger VM reset
> > to recover).
> 
> Yes, this is the extent of the implementation currently.
> 
> > The AER forward was proposed as a relatively simple opt-in mechanism
> > to avoid hitting RUN_STATE_INTERNAL_ERROR and be able to handle
> > errors for SR-IOV VFs using VFIO. For uncorrectable errors with
> > SR-IOV, PF driver can be the entity that handles the device recovery.
> > The VF driver needs to stop using the device until it is recovered by
> > PF, and reinitialize any state that was lost as part of the recovery
> > action. Forwarding AER would allow the VM to move forward without
> > introducing any new uAPI.
> 
> Interesting, nothing in the proposal suggested this was explicitly for
> SR-IOV VFs.
> 
> > If we would want to build a generic error recovery mechanism, we would
> > need to duplicate more of the err_handler_t callbacks as VFIO uAPI.
> >
> > If the recovery is handled by other entity running on the host (VFIO
> > variant? Or for SR-IOV, the PF driver), we could also go into a
> > different runstate (RUN_STATE_PAUSED? something that doesn't require
> > full VM reset) and add a VFIO uAPI that can propagate the .resume()
> > into userspace.
> 
> And a .slot_reset() to actually restore state if PF is reset (we
> currently rely on that happening on .release()), and we'll also need to
> teardown things like interrupts (also currently reliant on release).
> 
> > If we would want the SW running inside the VM to handle the
> > recovery... it becomes more involved, and it requires a more
> > substantial solution that wasn't meant to be addressed here.
> 
> Note that the current error eventfd is triggered on any uncorrected
> error, fatal or non-fatal.  How does the VM know which it is?  There
> are certainly hacks we can do to limp along a little further with the
> current limited implementation, but nothing that seems to actually get
> us closer to a general purpose, supportable feature.
> 
> > So - we're letting the VMM continue but we're also informing it that
> > something went wrong with the device, so that the SW running inside
> > the VM can do its own action.
> > If it's a virtual function, the driver can potentially fully recover
> > after the host does its thing.
> > If it's a regular native function, it will probably need to go into
> > PCI_ERS_RESULT_DISCONNECT, which, for most usecases is still a better
> > scenario then having to reset the entire VM.
> 
> We've been discussing this internally as well.  The VM can perform its
> own recovery, resetting and re-initializing devices from a VM
> perspective, but it can't "handle" the host recovery process.  In fact,
> I think at best it could pause and observe the host process, resuming
> and injecting an error after host recovery completes.
> 
> > If we would want to absolutely make sure that the users are not able
> > to issue any IO across error recovery, we can probably unmap (or
> > zero-map) the BARs at VFIO driver level, similar to what's done
> > during reset.
> 
> Yes, the shape I imagined would immediately block all access to the
> device on .err_detected(), zap BARs, fail access, and signal the
> existing error eventfd.  The VMM would need to handle the potential
> sigbus it might see before noticing the eventfd, determine whether it
> maps to the mmio space of a device and call a new device feature ioctl
> to test whether the device is in an error state (hand waving how QEMU
> gets from the trap handler to a point where the VM is paused and we can
> poll an ioctl).  Flag bits on the GET of that feature ioctl might
> indicate Fatal/Non-Fatal, InProgress, DeviceReset, and Failed.  Perhaps
> even a sequence number and/or SET on the feature ioctl might implement
> a W1C style acknowledgment - some mechanism by which the user can track
> whether they've missed an event.  QEMU could autonomously perform a
> surprise hot-unplug on Fatal or Failed, or otherwise pause the VM for
> InProgress to clear and inject an appropriate error when it does.
> APEI/GHES handling in the guest might be particularly useful to
> indicate to the VM whether a device reset has already been performed.
> In fact, the whole process doesn't seem too dissimilar to firmware
> first error handling that runs underneath the host OS on bare metal.

As Alex mentioned, we have been discussing this internally. I was
planning to look further into the observer-based approach outlined
above, including the required vfio-pci and QEMU changes.

However, I will be on PTO for the next two weeks, so I will only be
able to pick this up after I return. By all means, please go ahead
if this is urgent for you. I can catch up with any progress when
I am back.

Thanks,
Shameer