Re: SSD surprise removal leads to long wait inside pci_dev_wait() and FLR 65s timeout

[Alex Williamson <alex.williamson@redhat.com>]

On Mon, 3 Jun 2019 14:17:36 -0700
JD Zheng <jiandong.zheng@broadcom.com> wrote:

> On 6/2/19 6:40 PM, Alex Williamson wrote:
> > On Sun, 2 Jun 2019 19:44:14 -0500
> > Bjorn Helgaas <helgaas@kernel.org> wrote:
> >   
> >> [+cc Alex, Lukas]
> >>
> >> On Fri, May 31, 2019 at 09:55:20AM -0700, JD Zheng wrote:  
> >>> Hello,
> >>>
> >>> I am running DPDK 18.11+SPDK 19.04 with v5.1 kernel. DPDK/SPDK uses SSD vfio
> >>> devices and after running SPDK's nvmf_tgt, unplugging a SSD cause kernel to
> >>> print out following:
> >>> [  105.426952] vfio-pci 0000:04:00.0: not ready 2047ms after FLR; waiting
> >>> [  107.698953] vfio-pci 0000:04:00.0: not ready 4095ms after FLR; waiting
> >>> [  112.050960] vfio-pci 0000:04:00.0: not ready 8191ms after FLR; waiting
> >>> [  120.498953] vfio-pci 0000:04:00.0: not ready 16383ms after FLR; waiting
> >>> [  138.418957] vfio-pci 0000:04:00.0: not ready 32767ms after FLR; waiting
> >>> [  173.234953] vfio-pci 0000:04:00.0: not ready 65535ms after FLR; giving up
> >>>
> >>> Looks like it is a PCI hotplug racing condition between DPDK's
> >>> eal-intr-thread thread and kernel's pciehp thread. And it causes lockup in
> >>> pci_dev_wait() at kernel side.
> >>>
> >>> When SSD is removed, eal-intr-thread immediately receives
> >>> RTE_INTR_HANDLE_ALARM and handler calls rte_pci_detach_dev() and at kernel
> >>> side vfio_pci_release() is triggered to release this vfio device, which
> >>> calls pci_try_reset_function(), then _pci_reset_function_locked().
> >>> pci_try_reset_function acquires the device lock but
> >>> _pci_reset_function_locked() doesn't return, therefore lock is NOT released.  
> > 
> > To what extent does vfio-pci need to learn about surprise hotplug?  My
> > expectation is that the current state of the code would only support
> > cooperative hotplug.  When a device is surprise removed, what backs a
> > user's mmaps?  AIUI, we don't have a revoke interface to invalidate
> > these.  We should probably start with an RFE or some development effort
> > to harden vfio-pci for surprise hotplug, it's not surprising it doesn't
> > just work TBH.  Thanks,
> > 
> > Alex
> > 
> >   
> 
> I did see other issues that DPDK unmap vifo device memory was stuck due 
> to surprise removal.
> 
> Are you saying that vfio-pci surprise removal is not fully implemented yet?

Has not even been evaluated for that use case afaik.  You are in
unknown and expected broken territory.  If this is an area you'd like
to contribute, great, but expect more than a bug fix here and there.
Thanks,

Alex