Re: [PATCH] vfio/pci: Restore standard PCI config space in .slot_reset()

[Alex Williamson <alex@shazbot.org>]

[Cc +Shameer]

On Mon, 17 Aug 2026 17:21:23 -0600
Keith Busch <kbusch@kernel.org> wrote:

> On Mon, Aug 17, 2026 at 03:50:06PM -0600, Alex Williamson wrote:
> > > In scenarios where userspace simply needs the underlying hardware
> > > restored to its known-good baseline state post-reset, having vfio-pci
> > > implement .slot_reset() and leverage the existing PCI core snapshot via
> > > pci_restore_state() bridges this gap cleanly.  
> > 
> > Does it though?  Even for a simple reset to initial state we need to
> > prevent the host and guest stepping on each other across the reset as
> > well as tear down user modified state, like interrupts.  
> 
> Specifically considering host-guest interactions, I don't think this
> scenario is handled at all. After 14 years, this is the state of QEMU:
> 
>   static void vfio_err_notifier_handler(void *opaque)
>   {
>       VFIOPCIDevice *vdev = opaque;
>   
>       if (!event_notifier_test_and_clear(&vdev->err_notifier)) {
>           return;
>       }
>   
>       /*
>        * TBD. Retrieve the error details and decide what action
>        * needs to be taken. One of the actions could be to pass
>        * the error to the guest and have the guest driver recover
>        * from the error. This requires that PCIe capabilities be
>        * exposed to the guest. For now, we just terminate the
>        * guest to contain the error.
>        */
>   
>       error_report("%s(%s) Unrecoverable error detected. Please collect any data possible and then kill the guest", __func__, vdev->vbasedev.name);
>   
>       vm_stop(RUN_STATE_INTERNAL_ERROR);
>   }
> 
> Is there another common VMM that actually does something useful to
> continue from this event?

Not that I'm aware of, the kernel interface really isn't designed for
recovery, it's designed only to notify.
 
> Outside virtualization, I'm more interested in enabling user space
> DPDK-like drivers. I don't want to break anyone, so starting small here:
> restoring the config space to the baseline before the device was handed
> to a user space driver feels right.

But we have no hand-back-to-userspace mechanism currently.  Alone, it's
certainly a step towards letting the device run again, but we really
need more uAPI defined to provide coordination.

> > > A few questions I'd like to put to the list:
> > > 1. Since the PCI core already takes responsibility for holding the early 
> > >    config space checkpoint, is calling pci_restore_state() during
> > >    .slot_reset() the appropriate place for vfio-pci to apply it, or
> > >    should this be explicitly driven/triggered via a VFIO ioctl?
> > > 2. Does returning PCI_ERS_RESULT_RECOVERED here create subtle state
> > >    issues if userspace directly modified config space registers that
> > >    were not captured in pdev->saved_config_space?
> > > 3. Should we pair this with an explicit "link restored / reset complete"
> > >    eventfd notification so userspace knows exactly when it is safe to
> > >    resume access?
> > > 
> > > Appreciate any feedback or historical context on how VFIO and PCI error
> > > recovery should interact here.  
> > 
> > Certainly the host saved state doesn't take into account user
> > manipulation of the device since the last snapshot.   
> 
> Yeah, the kernel emits an eventfd that an error occured. The user side
> should have some baseline from which to proceed. It feels outside the
> scope of user space to save and restore such low level and early
> initialization things like the PCI BAR config space.

There's a fair bit of config space the user cannot write, particularly
BARs, so a VM has an obligation to restore the virtualized BARs to make
a coherent view of the device, but a userspace driver can't effect a
meaningful value change of the physical BAR register anyway.

> Also consider that the user space side may not have even been
> initialized at the time a PCIe error occured. What happens then?

I'd tend to think a userspace driver would consider aborting if the
device is triggering errors before they've even touched it.  Closing
the device writes back the state saved on open.

> > But also, vfio-pci error handling is currently limited to generating
> > an event when a non-recoverable error has occurred.  I don't think we
> > can nudge it forward in any meaningful way by implementing a
> > .slot_reset to restore a prior host snapshot.  That misses the user
> > modified state, coordination across error handling, and may not even
> > match the hand-off state of the device to the user.  
> 
> I totally agree. Lacking a notification, user space can at best poll
> something specific to their device, but it'd be better to generically
> coordinate this sequence with the kernel's error handling.
> 
> Would it be acceptable to introduce additional eventfd's for each part
> in the pcie error handling? Privately, I've proposed and tested the
> user-space component to quiesce on .error_detected, then start from
> scratch after the .slot_reset. But I still need something to restore the
> config space, and I feel kernel is the right place to do it.

Absolutely the kernel needs to provide some restore of the device,
especially where the user doesn't have access.  We need some mechanism
for the user to observe the host recovery and know when it can access
the device again.  I provided my high level vision of that in the link
I previously shared.  Shameer has also been looking into this and can
share his plans.

I might caution tying the uAPI too closely with the kernel recovery
implementation, for example eventfds mapped to each part of the
internal recovery sequence.  I think userspace needs to be an observer
to the kernel recovery, not a participant.  You're already doing
something similar with quiesce on error, but we need some mechanism to
know when the device is reachable again.  Recovery may also not be
successful, so an eventfd-only mechanism expands into multiple eventfds
to report success vs failure.  The VM use case probably wants a report
of the error, so it might be easier to have userspace poll an ioctl on
error to know when to continue or abort the device.  Thanks,

Alex