From mboxrd@z Thu Jan  1 00:00:00 1970
Content-Type: multipart/mixed; boundary="===============4116342309233489537=="
MIME-Version: 1.0
From: Harris, James R <james.r.harris at intel.com>
Subject: Re: [SPDK] PCIe hotplug support using VFIO for NVMf
Date: Tue, 08 May 2018 13:36:01 +0000
Message-ID: <9EDDCD9E-CE95-4E0D-8DB5-9DFD6B64C22C@intel.com>
In-Reply-To: b7645f0082eaf0be49aef2faef63e38b@mail.gmail.com
List-ID: <spdk@lists.01.org>
To: spdk@lists.01.org

--===============4116342309233489537==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable

Hi Jitendra,

There=E2=80=99s a loop at the end of _nvme_pcie_hotplug_monitor that is sup=
posed to work around this circular dependency.  It periodically reads a reg=
ister on the device to check if it=E2=80=99s been removed.  This loop was a=
dding prior to the v18.01 release.  Can you instrument that loop to see if =
it is triggering?

Thanks,

-Jim


On 5/8/18, 3:42 AM, "SPDK on behalf of Jitendra Bhivare" <spdk-bounces(a)li=
sts.01.org on behalf of jitendra.bhivare(a)broadcom.com> wrote:

    > For this to work I think we need an eventfd mechanism to notify SPDK =
of
    device removal from VFIO.
    Or a better approach, to generate KOBJ_CHANGE for vfio-pci driver kobje=
ct.
    =

    From: Jitendra Bhivare [mailto:jitendra.bhivare(a)broadcom.com]
    Sent: Tuesday, May 8, 2018 12:11 PM
    To: 'spdk(a)lists.01.org' <spdk(a)lists.01.org>
    Subject: PCIe hotplug support using VFIO for NVMf
    =

    Hi All,
    =

    I am trying to use PCIe hotplug feature in SPDK v18.01 (with DPDK v17.1=
1)
    for NVMf by setting in conf file HotplugEnable to Yes. It is not workin=
g.
    All the IOs get stuck on initiator side and nvmf_tgt does not even resp=
ond
    to nvme discovery query after that.
    =

    nvmf_tgt opens a netlink socket to listen on KOBJ events. Using VFIO
    claimed NVMe PCIe devices, when a PCIe device is removed vfio_pci_remove
    waits for all references to the device added in IOMMU group to be dropp=
ed
    in vfio_del_group_dev.
    =

    This reference will only be dropped after SPDK unloads the NVME PCIe
    driver. For that to happen it is waiting for the KOBJ events. KOBJ even=
ts
    won't happen till vfio_pci_remove releases the device.
    =

    So we kinda reached a deadlock with circular dependency on release of t=
he
    device.
    =

    Can someone please explain how this feature is working?
    =

    For this to work I think we need an eventfd mechanism to notify SPDK of
    device removal from VFIO.
    I am trying to integrate such a thing bit of a redundant approach made
    specifically for VFIO devices.
    =

    Please do let me know if we have better option or working on better
    approach to make this work.
    =

    Thanks,
    =

    JB
    _______________________________________________
    SPDK mailing list
    SPDK(a)lists.01.org
    https://lists.01.org/mailman/listinfo/spdk
    =



--===============4116342309233489537==--