Re: [SPDK] PCIe hotplug support using VFIO for NVMf

[Harris, James R <james.r.harris at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 2383 bytes --]

Hi Jitendra,

There’s a loop at the end of _nvme_pcie_hotplug_monitor that is supposed to work around this circular dependency.  It periodically reads a register on the device to check if it’s been removed.  This loop was adding prior to the v18.01 release.  Can you instrument that loop to see if it is triggering?

Thanks,

-Jim


On 5/8/18, 3:42 AM, "SPDK on behalf of Jitendra Bhivare" <spdk-bounces(a)lists.01.org on behalf of jitendra.bhivare(a)broadcom.com> wrote:

    > For this to work I think we need an eventfd mechanism to notify SPDK of
    device removal from VFIO.
    Or a better approach, to generate KOBJ_CHANGE for vfio-pci driver kobject.
    
    From: Jitendra Bhivare [mailto:jitendra.bhivare(a)broadcom.com]
    Sent: Tuesday, May 8, 2018 12:11 PM
    To: 'spdk(a)lists.01.org' <spdk(a)lists.01.org>
    Subject: PCIe hotplug support using VFIO for NVMf
    
    Hi All,
    
    I am trying to use PCIe hotplug feature in SPDK v18.01 (with DPDK v17.11)
    for NVMf by setting in conf file HotplugEnable to Yes. It is not working.
    All the IOs get stuck on initiator side and nvmf_tgt does not even respond
    to nvme discovery query after that.
    
    nvmf_tgt opens a netlink socket to listen on KOBJ events. Using VFIO
    claimed NVMe PCIe devices, when a PCIe device is removed vfio_pci_remove
    waits for all references to the device added in IOMMU group to be dropped
    in vfio_del_group_dev.
    
    This reference will only be dropped after SPDK unloads the NVME PCIe
    driver. For that to happen it is waiting for the KOBJ events. KOBJ events
    won't happen till vfio_pci_remove releases the device.
    
    So we kinda reached a deadlock with circular dependency on release of the
    device.
    
    Can someone please explain how this feature is working?
    
    For this to work I think we need an eventfd mechanism to notify SPDK of
    device removal from VFIO.
    I am trying to integrate such a thing bit of a redundant approach made
    specifically for VFIO devices.
    
    Please do let me know if we have better option or working on better
    approach to make this work.
    
    Thanks,
    
    JB
    _______________________________________________
    SPDK mailing list
    SPDK(a)lists.01.org
    https://lists.01.org/mailman/listinfo/spdk