Re: [SPDK] PCIe hotplug support using VFIO for NVMf

[Jitendra Bhivare <jitendra.bhivare at broadcom.com>]

[-- Attachment #1: Type: text/plain, Size: 3276 bytes --]

Thanks a lot Jim.

Missed that. In our platform CSTS read value is 0 after surprise hotunplug.
Adding !csts.bits.rdy check makes hotplug working fine.
I think we would still need KOBJ event for controlled hotunplug done by
echo'ing in remove sysfs entry.
For that we will have to send something like KOBJ_CHANGE/KOBJ_UNBIND to SPDK
and parse it.

Regards,

JB

> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
> Sent: Tuesday, May 8, 2018 7:06 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] PCIe hotplug support using VFIO for NVMf
>
> Hi Jitendra,
>
> There’s a loop at the end of _nvme_pcie_hotplug_monitor that is supposed
> to work around this circular dependency.  It periodically reads a register
> on
> the device to check if it’s been removed.  This loop was adding prior to
> the
> v18.01 release.  Can you instrument that loop to see if it is triggering?
>
> Thanks,
>
> -Jim
>
>
> On 5/8/18, 3:42 AM, "SPDK on behalf of Jitendra Bhivare" <spdk-
> bounces(a)lists.01.org on behalf of jitendra.bhivare(a)broadcom.com> wrote:
>
>     > For this to work I think we need an eventfd mechanism to notify SPDK
> of
>     device removal from VFIO.
>     Or a better approach, to generate KOBJ_CHANGE for vfio-pci driver
> kobject.
>
>     From: Jitendra Bhivare [mailto:jitendra.bhivare(a)broadcom.com]
>     Sent: Tuesday, May 8, 2018 12:11 PM
>     To: 'spdk(a)lists.01.org' <spdk(a)lists.01.org>
>     Subject: PCIe hotplug support using VFIO for NVMf
>
>     Hi All,
>
>     I am trying to use PCIe hotplug feature in SPDK v18.01 (with DPDK
> v17.11)
>     for NVMf by setting in conf file HotplugEnable to Yes. It is not
> working.
>     All the IOs get stuck on initiator side and nvmf_tgt does not even
> respond
>     to nvme discovery query after that.
>
>     nvmf_tgt opens a netlink socket to listen on KOBJ events. Using VFIO
>     claimed NVMe PCIe devices, when a PCIe device is removed
> vfio_pci_remove
>     waits for all references to the device added in IOMMU group to be
> dropped
>     in vfio_del_group_dev.
>
>     This reference will only be dropped after SPDK unloads the NVME PCIe
>     driver. For that to happen it is waiting for the KOBJ events. KOBJ
> events
>     won't happen till vfio_pci_remove releases the device.
>
>     So we kinda reached a deadlock with circular dependency on release of
> the
>     device.
>
>     Can someone please explain how this feature is working?
>
>     For this to work I think we need an eventfd mechanism to notify SPDK
> of
>     device removal from VFIO.
>     I am trying to integrate such a thing bit of a redundant approach made
>     specifically for VFIO devices.
>
>     Please do let me know if we have better option or working on better
>     approach to make this work.
>
>     Thanks,
>
>     JB
>     _______________________________________________
>     SPDK mailing list
>     SPDK(a)lists.01.org
>     https://lists.01.org/mailman/listinfo/spdk
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk