Re: [SPDK] Handling of physical disk removals

[Harris, James R <james.r.harris at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 2946 bytes --]

Hi Baruch,

Thanks for raising this issue – there are absolutely changes that SPDK needs to make here.

Can you describe your code path a bit more?  Or let me try to guess and you can tell me where I’m wrong.


1)       You’re using spdk_nvme_probe() with a NULL trid and a remove_cb handler to detect physically removed devices.

2)       In your remove_cb handler, you call spdk_nvme_detach().

The SPDK bdev nvme module doesn’t call spdk_nvme_detach() in its remove_cb which is why the SPDK automated tests don’t run into this issue.  But I don’t this is correct – we should be calling spdk_nvme_detach() at some point to clean up any allocated resources.  It needs to make sure any associate IO channels are freed up first (to avoid racing between the remove callback and different threads submitting IO to that removed controller).

Could you file this as a bug in github?  Please add any additional details on how you’re hitting this issue if it’s different than what I’ve guessed above.

https://github.com/spdk/spdk/issues

Thanks,

Jim


From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Baruch Even <baruch(a)weka.io>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org>
Date: Wednesday, May 23, 2018 at 1:44 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Handling of physical disk removals

Hi,
I'm using spdk for local nvme through the nvme interface, I find that physical disk removals are not handled properly for my use case and wonder if others see it that way as well and if there is an intention to fix this.
Our system uses long running processes that control one or more disks at a time, if a disk fails it may drop completely from the pcie bus and it will also look like that if the disk is physically removed (say a technician mistakes the disk that he should replace).
The problem that I see is that spdk doesnt consider a device completely disappearing from the bus and will try to release the io qpair by sending the delete io sq and delete io cq commands, both of these will never get an answer (the device is not on the pcie device anymore) and there is no timeout logic in that code path. This means two things, the process will halt forever and there is an effective memory leak which currently means that we need to restart the process. Now, our system is resilient enough that restarting the process is not a big deal but it is a very messy way to go about handlign a physical drive removal.

Have others seen this behavior? Does it bother others?
For my own use I put a timeout in there of a few seconds and that solves it for me.

Baruch Even

--
[https://docs.google.com/uc?export=download&id=1w6mlhCJZRlvqVOKRoulFtalU3TMY41VY&revid=0Bw_6cJeSSMVJR2JMQzdxV3VqVi9IWFNDM1FVcnFoRlc2NkJzPQ]
Baruch Even, Software Developer

E  baruch(a)weka.io<mailto:liran(a)weka.io>
www.weka.io<http://www.weka.io>

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 11409 bytes --]