Re: [PATCH] PCI: hotplug: Allow marking devices as disconnected during bind/unbind

[Lukas Wunner <lukas@wunner.de>]

On Fri, Jan 20, 2023 at 10:41:13AM +0100, Christian König wrote:
> Am 20.01.23 um 10:19 schrieb Lukas Wunner:
> > On surprise removal, pciehp_unconfigure_device() and acpiphp's
> > trim_stale_devices() call pci_dev_set_disconnected() to mark removed
> > devices as permanently offline.  Thereby, the PCI core and drivers know
> > to skip device accesses.
> > 
> > However pci_dev_set_disconnected() takes the device_lock and thus waits
> > for a concurrent driver bind or unbind to complete.  As a result, the
> > driver's ->probe and ->remove hooks have no chance to learn that the
> > device is gone.
> 
> Who is reading dev->error_state in this situation and especially do we have
> the necessary read barrier in place?
> 
> Alternatively if this is just opportunistically we should document that
> somehow.

dev->error_state is read by pci_dev_is_disconnected() and by various
drivers.  None of them has a specific read barrier AFAICS or is using
the device_lock to protect read access.

For pci_dev_is_disconnected(), the read is indeed opportunistic.
It's an optimization that prevents the kernel from performing
a lot of non-posted requests to a removed device and waiting
for them to time out.  Or worse, having them be received by a
replacement device.


> > That doesn't make any sense, so drop the device_lock and instead use
> > atomic xchg() and cmpxchg() operations to update the device state.
> 
> You use xchg() instead of WRITE_ONCE() for the memory barrier here?

xchg() implies a full memory barrier to ensure that the new state is
immediately visible to all CPUs.  WRITE_ONCE() would be weaker,
it's just a compiler barrier.

The desire to immediately make the state change visible is why we
absolutely do not want locking here.


> > As a byproduct, an AB-BA deadlock reported by Anatoli is fixed which
> > occurs on surprise removal with AER concurrently performing a bus reset.
> 
> Well this byproduct is probably the main fix in this patch. I'm wondering
> why lockdep didn't complained about that more drastically in our testing.

Well I guess I could rephrase the commit message if you feel I'm burying
the lede.

Thanks,

Lukas