Locking between vfio hot-remove and pci sysfs sriov_numvfs

[Jim Harris <jim.harris@samsung.com>]

I am seeing a deadlock using SPDK with hotplug detection using vfio-pci
and an SR-IOV enabled NVMe SSD. It is not clear if this deadlock is intended
or if it's a kernel bug.

Note: SPDK uses DPDK's PCI device enumeration framework, so I'll reference
both SPDK and DPDK in this description.

DPDK registers an eventfd with vfio for hotplug notifications. If the associated
device is removed (i.e. write 1 to its pci sysfs remove entry), vfio
writes to the eventfd, requesting DPDK to release the device. It does this
while holding the device_lock(), and then waits for completion.

DPDK gets the notification, and passes it up to SPDK. SPDK does not release
the device immediately. It has some asynchronous operations that need to be
performed first, so it will release the device a bit later.

But before the device is released, SPDK also triggers DPDK to do a sysfs scan
looking for newly inserted devices. Note that the removed device is not
completely removed yet from kernel PCI perspective - all of its sysfs entries
are still available, including sriov_numvfs.

DPDK explicitly reads sriov_numvfs to see if the device is SR-IOV capable.
SPDK itself doesn't actually use this value, but it is part of the scan
triggered by SPDK and directly leads to the deadlock. sriov_numvfs_show()
deadlocks because it tries to hold device_lock() while reading the pci
device's pdev->sriov->num_VFs.

We're able to workaround this in SPDK by deferring the sysfs scan if
a device removal is in process. And maybe that is what we are supposed to
be doing, to avoid this deadlock?

Reference to SPDK issue, for some more details (plus simple repro stpes for
anyone already familiar with SPDK): https://github.com/spdk/spdk/issues/3205