[SPDK] NVMe device reinsertion not detected

[SPDK] NVMe device reinsertion not detected

[Baruch Even]

[-- Attachment #1: Type: text/plain, Size: 1170 bytes --]

Hi,

I'm using spdk 16.12 and dpdk 16.11 and am having an issue with NVMe device
reinsertion that is not getting detected and I wanted to check with you
about the current state of device hot plug and hot removal as I've seen at
ticket that mentions there are known issues with these.

Currently we have the devices working at startup by detecting the already
bound devices at startup and that works just fine. When I then go on to
test device removal the removal gets detected fine when I call the probe
function periodically[1]. If I reinsert the device and bind it to the
uio_pci_generic driver it is not being detected by spdk and is not added
back, at least there is no probe/attach callbacks getting called and I
can't see any traces from dpdk showing up.

If I restart the process the device will get detected and taken back as
usual and it can serve as a temporary workaround but it is not a good
overall approach.

Is this a known issue?
What can be done about this?

Baruch

[1] It would be nice to expose this file descriptor to allow
select/poll/epoll to be used to call an event instead of periodic polls for
this very rare event.

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 1307 bytes --]

Re: [SPDK] NVMe device reinsertion not detected

[Chang, Cunyin]

[-- Attachment #1: Type: text/plain, Size: 2001 bytes --]

Hi Baruch,

The case you mentioned is not what we expected, we suppose the driver should detect the
Hot insert event when you insert a new device into system and bind it to uio_pci_generic driver, I suggest you use the latest SPDK
Version and DPDK17.02, please also make sure the CONFIG_DEBUG=y in CONFIG file before you compile the SPDK, if SPDK get new
Event, you will find the dump information: “add nvme address:”, which means SPDK detect one event from uio_pci_generic driver.
If not, maybe something wrong when you bind the device to uio_pci_generic driver. Please also provide some details steps when you test
Hotplug.

Thanks,
Cunyin

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Baruch Even
Sent: Thursday, April 20, 2017 3:05 PM
To: spdk(a)lists.01.org
Subject: [SPDK] NVMe device reinsertion not detected

Hi,
I'm using spdk 16.12 and dpdk 16.11 and am having an issue with NVMe device reinsertion that is not getting detected and I wanted to check with you about the current state of device hot plug and hot removal as I've seen at ticket that mentions there are known issues with these.
Currently we have the devices working at startup by detecting the already bound devices at startup and that works just fine. When I then go on to test device removal the removal gets detected fine when I call the probe function periodically[1]. If I reinsert the device and bind it to the uio_pci_generic driver it is not being detected by spdk and is not added back, at least there is no probe/attach callbacks getting called and I can't see any traces from dpdk showing up.
If I restart the process the device will get detected and taken back as usual and it can serve as a temporary workaround but it is not a good overall approach.

Is this a known issue?
What can be done about this?
Baruch
[1] It would be nice to expose this file descriptor to allow select/poll/epoll to be used to call an event instead of periodic polls for this very rare event.

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 6308 bytes --]

Re: [SPDK] NVMe device reinsertion not detected

[Baruch Even]

[-- Attachment #1: Type: text/plain, Size: 2608 bytes --]

Thanks. I'll try it with the latest SPDK.

It was also suggested to me offlne to try and do a direct probe of the
devices which is something that is only supported from the latest spdk.

Baruch

On Fri, Apr 21, 2017 at 3:46 AM Chang, Cunyin <cunyin.chang(a)intel.com>
wrote:

> Hi Baruch,
>
>
>
> The case you mentioned is not what we expected, we suppose the driver
> should detect the
>
> Hot insert event when you insert a new device into system and bind it to
> uio_pci_generic driver, I suggest you use the latest SPDK
>
> Version and DPDK17.02, please also make sure the CONFIG_DEBUG=y in CONFIG
> file before you compile the SPDK, if SPDK get new
>
> Event, you will find the dump information: “add nvme address:”, which
> means SPDK detect one event from uio_pci_generic driver.
>
> If not, maybe something wrong when you bind the device to uio_pci_generic
> driver. Please also provide some details steps when you test
>
> Hotplug.
>
>
>
> Thanks,
>
> Cunyin
>
>
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Baruch Even
> *Sent:* Thursday, April 20, 2017 3:05 PM
> *To:* spdk(a)lists.01.org
> *Subject:* [SPDK] NVMe device reinsertion not detected
>
>
>
> Hi,
>
> I'm using spdk 16.12 and dpdk 16.11 and am having an issue with NVMe
> device reinsertion that is not getting detected and I wanted to check with
> you about the current state of device hot plug and hot removal as I've seen
> at ticket that mentions there are known issues with these.
>
> Currently we have the devices working at startup by detecting the already
> bound devices at startup and that works just fine. When I then go on to
> test device removal the removal gets detected fine when I call the probe
> function periodically[1]. If I reinsert the device and bind it to the
> uio_pci_generic driver it is not being detected by spdk and is not added
> back, at least there is no probe/attach callbacks getting called and I
> can't see any traces from dpdk showing up.
>
> If I restart the process the device will get detected and taken back as
> usual and it can serve as a temporary workaround but it is not a good
> overall approach.
>
>
>
> Is this a known issue?
>
> What can be done about this?
>
> Baruch
>
> [1] It would be nice to expose this file descriptor to allow
> select/poll/epoll to be used to call an event instead of periodic polls for
> this very rare event.
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 5930 bytes --]

Re: [SPDK] NVMe device reinsertion not detected

[Baruch Even]

[-- Attachment #1: Type: text/plain, Size: 2978 bytes --]

Hi,

Is it a requirement to use the exact DPDK version or can I use dpdk 16.11
with spdk 17.03? 'm having some problems and am wondering if this is a
supported version at all.

Thanks,
Baruch

On Sun, Apr 23, 2017 at 11:28 AM Baruch Even <baruch(a)weka.io> wrote:

> Thanks. I'll try it with the latest SPDK.
>
> It was also suggested to me offlne to try and do a direct probe of the
> devices which is something that is only supported from the latest spdk.
>
> Baruch
>
> On Fri, Apr 21, 2017 at 3:46 AM Chang, Cunyin <cunyin.chang(a)intel.com>
> wrote:
>
>> Hi Baruch,
>>
>>
>>
>> The case you mentioned is not what we expected, we suppose the driver
>> should detect the
>>
>> Hot insert event when you insert a new device into system and bind it to
>> uio_pci_generic driver, I suggest you use the latest SPDK
>>
>> Version and DPDK17.02, please also make sure the CONFIG_DEBUG=y in CONFIG
>> file before you compile the SPDK, if SPDK get new
>>
>> Event, you will find the dump information: “add nvme address:”, which
>> means SPDK detect one event from uio_pci_generic driver.
>>
>> If not, maybe something wrong when you bind the device to uio_pci_generic
>> driver. Please also provide some details steps when you test
>>
>> Hotplug.
>>
>>
>>
>> Thanks,
>>
>> Cunyin
>>
>>
>>
>> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Baruch
>> Even
>> *Sent:* Thursday, April 20, 2017 3:05 PM
>> *To:* spdk(a)lists.01.org
>> *Subject:* [SPDK] NVMe device reinsertion not detected
>>
>>
>>
>> Hi,
>>
>> I'm using spdk 16.12 and dpdk 16.11 and am having an issue with NVMe
>> device reinsertion that is not getting detected and I wanted to check with
>> you about the current state of device hot plug and hot removal as I've seen
>> at ticket that mentions there are known issues with these.
>>
>> Currently we have the devices working at startup by detecting the already
>> bound devices at startup and that works just fine. When I then go on to
>> test device removal the removal gets detected fine when I call the probe
>> function periodically[1]. If I reinsert the device and bind it to the
>> uio_pci_generic driver it is not being detected by spdk and is not added
>> back, at least there is no probe/attach callbacks getting called and I
>> can't see any traces from dpdk showing up.
>>
>> If I restart the process the device will get detected and taken back as
>> usual and it can serve as a temporary workaround but it is not a good
>> overall approach.
>>
>>
>>
>> Is this a known issue?
>>
>> What can be done about this?
>>
>> Baruch
>>
>> [1] It would be nice to expose this file descriptor to allow
>> select/poll/epoll to be used to call an event instead of periodic polls for
>> this very rare event.
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>>
>

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 6632 bytes --]

Re: [SPDK] NVMe device reinsertion not detected

[Chang, Cunyin]

[-- Attachment #1: Type: text/plain, Size: 3230 bytes --]

Hi Baruch,

Let’s keep using DPDK 17.02 + SPDK 17.03, we have some modification from DPDK 16.11 to DPDK 17.02 for the pci devices probe.

Thanks,
Cunyin

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Baruch Even
Sent: Sunday, April 23, 2017 7:46 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] NVMe device reinsertion not detected

Hi,
Is it a requirement to use the exact DPDK version or can I use dpdk 16.11 with spdk 17.03? 'm having some problems and am wondering if this is a supported version at all.
Thanks,
Baruch

On Sun, Apr 23, 2017 at 11:28 AM Baruch Even <baruch(a)weka.io<mailto:baruch(a)weka.io>> wrote:
Thanks. I'll try it with the latest SPDK.
It was also suggested to me offlne to try and do a direct probe of the devices which is something that is only supported from the latest spdk.

Baruch

On Fri, Apr 21, 2017 at 3:46 AM Chang, Cunyin <cunyin.chang(a)intel.com<mailto:cunyin.chang(a)intel.com>> wrote:
Hi Baruch,

The case you mentioned is not what we expected, we suppose the driver should detect the
Hot insert event when you insert a new device into system and bind it to uio_pci_generic driver, I suggest you use the latest SPDK
Version and DPDK17.02, please also make sure the CONFIG_DEBUG=y in CONFIG file before you compile the SPDK, if SPDK get new
Event, you will find the dump information: “add nvme address:”, which means SPDK detect one event from uio_pci_generic driver.
If not, maybe something wrong when you bind the device to uio_pci_generic driver. Please also provide some details steps when you test
Hotplug.

Thanks,
Cunyin

From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Baruch Even
Sent: Thursday, April 20, 2017 3:05 PM
To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>
Subject: [SPDK] NVMe device reinsertion not detected

Hi,
I'm using spdk 16.12 and dpdk 16.11 and am having an issue with NVMe device reinsertion that is not getting detected and I wanted to check with you about the current state of device hot plug and hot removal as I've seen at ticket that mentions there are known issues with these.
Currently we have the devices working at startup by detecting the already bound devices at startup and that works just fine. When I then go on to test device removal the removal gets detected fine when I call the probe function periodically[1]. If I reinsert the device and bind it to the uio_pci_generic driver it is not being detected by spdk and is not added back, at least there is no probe/attach callbacks getting called and I can't see any traces from dpdk showing up.
If I restart the process the device will get detected and taken back as usual and it can serve as a temporary workaround but it is not a good overall approach.

Is this a known issue?
What can be done about this?
Baruch
[1] It would be nice to expose this file descriptor to allow select/poll/epoll to be used to call an event instead of periodic polls for this very rare event.
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 11326 bytes --]

Re: [SPDK] NVMe device reinsertion not detected

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 613 bytes --]

On Sun, 2017-04-23 at 11:45 +0000, Baruch Even wrote:
> Is it a requirement to use the exact DPDK version or can I use dpdk
> 16.11 with spdk 17.03? 'm having some problems and am wondering if
> this is a supported version at all.
> 

We try to support older DPDK releases as much as possible and the
default for new SPDK releases is that they are backward compatible with
all previous DPDK releases that we supported before. However, for a new
feature like hotplug both DPDK and SPDK are co-evolving support at the
same time, so you'll need to use the latest from both projects for the
best results.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 3274 bytes --]

Re: [SPDK] NVMe device reinsertion not detected

[Baruch Even]

[-- Attachment #1: Type: text/plain, Size: 1190 bytes --]

Thanks. I'll probably upgrade my DPDK then as well. For now I got the
system to work with only upgrading SPDK but to avoid possibly latent issues
I will upgrade DPDK as well. Our system is using DPDK for networking so it
requires more coordination to upgrade DPDK without affecting other parts
and features.

Baruch

On Mon, Apr 24, 2017 at 7:19 PM Walker, Benjamin <benjamin.walker(a)intel.com>
wrote:

> On Sun, 2017-04-23 at 11:45 +0000, Baruch Even wrote:
> > Is it a requirement to use the exact DPDK version or can I use dpdk
> > 16.11 with spdk 17.03? 'm having some problems and am wondering if
> > this is a supported version at all.
> >
>
> We try to support older DPDK releases as much as possible and the
> default for new SPDK releases is that they are backward compatible with
> all previous DPDK releases that we supported before. However, for a new
> feature like hotplug both DPDK and SPDK are co-evolving support at the
> same time, so you'll need to use the latest from both projects for the
> best results._______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 1624 bytes --]