Re: [SPDK] PCIe hotplug support using VFIO for NVMf

[Jitendra Bhivare <jitendra.bhivare at broadcom.com>]

[-- Attachment #1: Type: text/plain, Size: 3841 bytes --]

Using RPC to delete first, controlled hotunplug too works fine.

Thanks,

JB

> -----Original Message-----
> From: Jitendra Bhivare [mailto:jitendra.bhivare(a)broadcom.com]
> Sent: Wednesday, May 9, 2018 5:03 PM
> To: 'Storage Performance Development Kit' <spdk(a)lists.01.org>
> Subject: RE: [SPDK] PCIe hotplug support using VFIO for NVMf
>
> Thanks a lot Jim.
>
> Missed that. In our platform CSTS read value is 0 after surprise
> hotunplug.
> Adding !csts.bits.rdy check makes hotplug working fine.
> I think we would still need KOBJ event for controlled hotunplug done by
> echo'ing in remove sysfs entry.
> For that we will have to send something like KOBJ_CHANGE/KOBJ_UNBIND
> to SPDK and parse it.
>
> Regards,
>
> JB
>
> > -----Original Message-----
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris,
> > James R
> > Sent: Tuesday, May 8, 2018 7:06 PM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: Re: [SPDK] PCIe hotplug support using VFIO for NVMf
> >
> > Hi Jitendra,
> >
> > There’s a loop at the end of _nvme_pcie_hotplug_monitor that is
> > supposed to work around this circular dependency.  It periodically
> > reads a register on the device to check if it’s been removed.  This
> > loop was adding prior to the
> > v18.01 release.  Can you instrument that loop to see if it is
> > triggering?
> >
> > Thanks,
> >
> > -Jim
> >
> >
> > On 5/8/18, 3:42 AM, "SPDK on behalf of Jitendra Bhivare" <spdk-
> > bounces(a)lists.01.org on behalf of jitendra.bhivare(a)broadcom.com>
> wrote:
> >
> >     > For this to work I think we need an eventfd mechanism to notify
> > SPDK
> of
> >     device removal from VFIO.
> >     Or a better approach, to generate KOBJ_CHANGE for vfio-pci driver
> > kobject.
> >
> >     From: Jitendra Bhivare [mailto:jitendra.bhivare(a)broadcom.com]
> >     Sent: Tuesday, May 8, 2018 12:11 PM
> >     To: 'spdk(a)lists.01.org' <spdk(a)lists.01.org>
> >     Subject: PCIe hotplug support using VFIO for NVMf
> >
> >     Hi All,
> >
> >     I am trying to use PCIe hotplug feature in SPDK v18.01 (with DPDK
> v17.11)
> >     for NVMf by setting in conf file HotplugEnable to Yes. It is not
> > working.
> >     All the IOs get stuck on initiator side and nvmf_tgt does not even
> > respond
> >     to nvme discovery query after that.
> >
> >     nvmf_tgt opens a netlink socket to listen on KOBJ events. Using VFIO
> >     claimed NVMe PCIe devices, when a PCIe device is removed
> > vfio_pci_remove
> >     waits for all references to the device added in IOMMU group to be
> > dropped
> >     in vfio_del_group_dev.
> >
> >     This reference will only be dropped after SPDK unloads the NVME PCIe
> >     driver. For that to happen it is waiting for the KOBJ events. KOBJ
> > events
> >     won't happen till vfio_pci_remove releases the device.
> >
> >     So we kinda reached a deadlock with circular dependency on release
> > of
> the
> >     device.
> >
> >     Can someone please explain how this feature is working?
> >
> >     For this to work I think we need an eventfd mechanism to notify SPDK
> > of
> >     device removal from VFIO.
> >     I am trying to integrate such a thing bit of a redundant approach
> > made
> >     specifically for VFIO devices.
> >
> >     Please do let me know if we have better option or working on better
> >     approach to make this work.
> >
> >     Thanks,
> >
> >     JB
> >     _______________________________________________
> >     SPDK mailing list
> >     SPDK(a)lists.01.org
> >     https://lists.01.org/mailman/listinfo/spdk
> >
> >
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk