Re: [PATCH] vfio/pci: Restore standard PCI config space in .slot_reset()

[Alex Williamson <alex@shazbot.org>]

On Mon, 17 Aug 2026 12:51:10 -0700
Keith Busch <kbusch@meta.com> wrote:

> From: Keith Busch <kbusch@kernel.org>
> 
> Hi Alex, Bjorn, Linux-PCI, and VFIO,
> 
> I'd like to initiate, or maybe restart, a discussion regarding the
> current state of PCIe AER and DPC (Downstream Port Containment) recovery
> for devices bound to vfio-pci, potentially expanding to devices not
> bound to anything at all.
> 
> Currently, when a link reset or Secondary Bus Reset occurs due to an
> AER/DPC event, the kernel PCI error recovery service executes reset
> routines across affected downstream devices. Since vfio-pci does not
> implement the .slot_reset() callback in its pci_error_handlers struct,
> the standard PCI config space (BARs, Command register, MSI-X setup,
> DevCtl, etc.) for downstream devices is left entirely uninitialized
> after the link is brought back up.
> 
> Historically the PCI core introduced early pci_save_state()
> checkpointing prior to driver probe (e.g. commit 30d52f6385d8, "PCI:
> Save device state in pci_pm_init() before driver probe") precisely to
> ensure that the kernel maintains a valid baseline config space
> snapshot—even for driverless or stub-bound devices. However, that
> infrastructure was never fully connected to the error recovery callbacks
> for generic pass-through drivers.
> 
> For bare-metal userspace drivers, like custom user-level drivers, this
> creates an operational deadlock:
> 
> 1. The kernel PCI error handling triggers a bus/link reset, but leaves
>    standard PCI config space wiped back to power-on defaults.
> 2. The eventfd (VFIO_PCI_ERR_IRQ_INDEX) alerts userspace that an error occurred,
>    but gives no indication of when link recovery/reset completes.
> 3. Userspace attempting to re-initialize or access the device races against
>    the kernel's reset sequence or operates on zeroed/uninitialized config registers.
> 4. While QEMU originally included stubs intended to coordinate AER recovery with
>    guest OS drivers, that guest-host coordination infrastructure has remained
>    unimplemented for over a decade.
> 
> In scenarios where userspace simply needs the underlying hardware
> restored to its known-good baseline state post-reset, having vfio-pci
> implement .slot_reset() and leverage the existing PCI core snapshot via
> pci_restore_state() bridges this gap cleanly.

Does it though?  Even for a simple reset to initial state we need to
prevent the host and guest stepping on each other across the reset as
well as tear down user modified state, like interrupts.

> We recently saw related discussions around VFIO error recovery on s390x
> (https://lore.kernel.org/linux-pci/20260603182415.2324-1-alifm@linux.ibm.com/#t),
> where platform-specific mechanisms had to be wired up to communicate PCI
> errors to userspace. However, for standard PCIe setups relying on
> generic kernel error recovery (pcie_do_recovery), vfio-pci still lacks
> the fundamental .slot_reset() handling needed to restore standard PCI
> config space after recoverying from a fatal error.

The s390x series preempts recovery for assigned devices and only
provides an extra reporting channel through vfio-pci.  I don't know if
that's a model we can (or want to) copy, but may work for them given the
underlying hypervisor virtualization of the device.

> ---
> A few questions I'd like to put to the list:
> 1. Since the PCI core already takes responsibility for holding the early 
>    config space checkpoint, is calling pci_restore_state() during
>    .slot_reset() the appropriate place for vfio-pci to apply it, or
>    should this be explicitly driven/triggered via a VFIO ioctl?
> 2. Does returning PCI_ERS_RESULT_RECOVERED here create subtle state
>    issues if userspace directly modified config space registers that
>    were not captured in pdev->saved_config_space?
> 3. Should we pair this with an explicit "link restored / reset complete"
>    eventfd notification so userspace knows exactly when it is safe to
>    resume access?
> 
> Appreciate any feedback or historical context on how VFIO and PCI error
> recovery should interact here.

Certainly the host saved state doesn't take into account user
manipulation of the device since the last snapshot.  But also, vfio-pci
error handling is currently limited to generating an event when a
non-recoverable error has occurred.  I don't think we can nudge it
forward in any meaningful way by implementing a .slot_reset to restore
a prior host snapshot.  That misses the user modified state,
coordination across error handling, and may not even match the hand-off
state of the device to the user.

Forwarding AER to the guest and continuing came up in another thread
recently where I proposed[1] an approach we might use.  Effectively we
need to proactively block access to the device during the host error
handling progression while providing some observability of that
process and the resulting state of the device.  GHES/APEI is
potentially a model that this "host-first" error handling hides behind
for a guest.  Thanks,

Alex

[1]https://lore.kernel.org/all/20260707161234.23ed28db@nvidia.com/

> 
>  drivers/vfio/pci/vfio_pci_core.c | 8 ++++++++
>  1 file changed, 8 insertions(+)
> 
> diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
> index 3f11a9624b9c0..5b42430337e29 100644
> --- a/drivers/vfio/pci/vfio_pci_core.c
> +++ b/drivers/vfio/pci/vfio_pci_core.c
> @@ -2347,6 +2347,13 @@ pci_ers_result_t vfio_pci_core_aer_err_detected(struct pci_dev *pdev,
>  }
>  EXPORT_SYMBOL_GPL(vfio_pci_core_aer_err_detected);
>  
> +pci_ers_result_t vfio_pci_core_aer_slot_reset(struct pci_dev *pdev,
> +					      pci_channel_state_t state)
> +{
> +	pci_restore_state(pdev);
> +	return PCI_ERS_RESULT_RECOVERED;
> +}
> +
>  int vfio_pci_core_sriov_configure(struct vfio_pci_core_device *vdev,
>  				  int nr_virtfn)
>  {
> @@ -2419,6 +2426,7 @@ EXPORT_SYMBOL_GPL(vfio_pci_core_sriov_configure);
>  
>  const struct pci_error_handlers vfio_pci_core_err_handlers = {
>  	.error_detected = vfio_pci_core_aer_err_detected,
> +	.slot_reset = vfio_pci_core_aer_slot_reset,
>  };
>  EXPORT_SYMBOL_GPL(vfio_pci_core_err_handlers);
>