Re: [SPDK] PCIe hotplug support using VFIO for NVMf

[Jitendra Bhivare <jitendra.bhivare at broadcom.com>]

[-- Attachment #1: Type: text/plain, Size: 1574 bytes --]

> For this to work I think we need an eventfd mechanism to notify SPDK of
device removal from VFIO.
Or a better approach, to generate KOBJ_CHANGE for vfio-pci driver kobject.

From: Jitendra Bhivare [mailto:jitendra.bhivare(a)broadcom.com]
Sent: Tuesday, May 8, 2018 12:11 PM
To: 'spdk(a)lists.01.org' <spdk(a)lists.01.org>
Subject: PCIe hotplug support using VFIO for NVMf

Hi All,

I am trying to use PCIe hotplug feature in SPDK v18.01 (with DPDK v17.11)
for NVMf by setting in conf file HotplugEnable to Yes. It is not working.
All the IOs get stuck on initiator side and nvmf_tgt does not even respond
to nvme discovery query after that.

nvmf_tgt opens a netlink socket to listen on KOBJ events. Using VFIO
claimed NVMe PCIe devices, when a PCIe device is removed vfio_pci_remove
waits for all references to the device added in IOMMU group to be dropped
in vfio_del_group_dev.

This reference will only be dropped after SPDK unloads the NVME PCIe
driver. For that to happen it is waiting for the KOBJ events. KOBJ events
won't happen till vfio_pci_remove releases the device.

So we kinda reached a deadlock with circular dependency on release of the
device.

Can someone please explain how this feature is working?

For this to work I think we need an eventfd mechanism to notify SPDK of
device removal from VFIO.
I am trying to integrate such a thing bit of a redundant approach made
specifically for VFIO devices.

Please do let me know if we have better option or working on better
approach to make this work.

Thanks,

JB