[PATCH] nvme-core: flush namespace scanning work just before removing namespaces

[PATCH] nvme-core: flush namespace scanning work just before removing namespaces

[Sagi Grimberg]

nvme_stop_ctrl can be called also for reset flow and there is no need to
flush the scan_work as namespaces are not being removed. This can cause
deadlock in rdma, fc and loop drivers since nvme_stop_ctrl barriers
before controller teardown (and specifically I/O cancellation of the
scan_work itself) takes place, but the scan_work will be blocked anyways
so there is no need to flush it.

Instead, move scan_work flush to nvme_remove_namespaces() where it really
needs to flush.

Reported-by: Ming Lei <ming.lei at redhat.com>
Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
This is a stable candidate...

 drivers/nvme/host/core.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index c418b7a347e0..2e0571584e7a 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -3327,6 +3327,9 @@ void nvme_remove_namespaces(struct nvme_ctrl *ctrl)
 	struct nvme_ns *ns, *next;
 	LIST_HEAD(ns_list);
 
+	/* prevent racing with ns scanning */
+	flush_work(&ctrl->scan_work);
+
 	/*
 	 * The dead states indicates the controller was not gracefully
 	 * disconnected. In that case, we won't be able to flush any data while
@@ -3489,7 +3492,6 @@ void nvme_stop_ctrl(struct nvme_ctrl *ctrl)
 	nvme_mpath_stop(ctrl);
 	nvme_stop_keep_alive(ctrl);
 	flush_work(&ctrl->async_event_work);
-	flush_work(&ctrl->scan_work);
 	cancel_work_sync(&ctrl->fw_act_work);
 	if (ctrl->ops->stop_ctrl)
 		ctrl->ops->stop_ctrl(ctrl);
-- 
2.17.1

[PATCH] nvme-core: flush namespace scanning work just before removing namespaces

[Ewan D. Milne]

On Wed, 2018-11-21@15:17 -0800, Sagi Grimberg wrote:
> nvme_stop_ctrl can be called also for reset flow and there is no need to
> flush the scan_work as namespaces are not being removed. This can cause
> deadlock in rdma, fc and loop drivers since nvme_stop_ctrl barriers
> before controller teardown (and specifically I/O cancellation of the
> scan_work itself) takes place, but the scan_work will be blocked anyways
> so there is no need to flush it.
> 
> Instead, move scan_work flush to nvme_remove_namespaces() where it really
> needs to flush.
> 
> Reported-by: Ming Lei <ming.lei at redhat.com>
> Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
> ---
> This is a stable candidate...
> 
>  drivers/nvme/host/core.c | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> index c418b7a347e0..2e0571584e7a 100644
> --- a/drivers/nvme/host/core.c
> +++ b/drivers/nvme/host/core.c
> @@ -3327,6 +3327,9 @@ void nvme_remove_namespaces(struct nvme_ctrl *ctrl)
>  	struct nvme_ns *ns, *next;
>  	LIST_HEAD(ns_list);
>  
> +	/* prevent racing with ns scanning */
> +	flush_work(&ctrl->scan_work);
> +
>  	/*
>  	 * The dead states indicates the controller was not gracefully
>  	 * disconnected. In that case, we won't be able to flush any data while
> @@ -3489,7 +3492,6 @@ void nvme_stop_ctrl(struct nvme_ctrl *ctrl)
>  	nvme_mpath_stop(ctrl);
>  	nvme_stop_keep_alive(ctrl);
>  	flush_work(&ctrl->async_event_work);
> -	flush_work(&ctrl->scan_work);
>  	cancel_work_sync(&ctrl->fw_act_work);
>  	if (ctrl->ops->stop_ctrl)
>  		ctrl->ops->stop_ctrl(ctrl);

With this change, I can no longer reproduce the NVMe fabrics reset hangs
I was seeing (e.g. after a few hundred resets).  A longer-duration fabric
connection bounce test survived as well, this change looks good to me.

Thanks for fixing this, Sagi.

Tested-by: Ewan D. Milne <emilne at redhat.com>

[PATCH] nvme-core: flush namespace scanning work just before removing namespaces

[Keith Busch]

On Wed, Nov 21, 2018@03:17:37PM -0800, Sagi Grimberg wrote:
> nvme_stop_ctrl can be called also for reset flow and there is no need to
> flush the scan_work as namespaces are not being removed. This can cause
> deadlock in rdma, fc and loop drivers since nvme_stop_ctrl barriers
> before controller teardown (and specifically I/O cancellation of the
> scan_work itself) takes place, but the scan_work will be blocked anyways
> so there is no need to flush it.
> 
> Instead, move scan_work flush to nvme_remove_namespaces() where it really
> needs to flush.
> 
> Reported-by: Ming Lei <ming.lei at redhat.com>
> Signed-off-by: Sagi Grimberg <sagi at grimberg.me>

Looks good to me.

Reviewed-by: Keith Busch <keith.busch at intel.com>

> --- a/drivers/nvme/host/core.c
> +++ b/drivers/nvme/host/core.c
> @@ -3327,6 +3327,9 @@ void nvme_remove_namespaces(struct nvme_ctrl *ctrl)
>  	struct nvme_ns *ns, *next;
>  	LIST_HEAD(ns_list);
>  
> +	/* prevent racing with ns scanning */
> +	flush_work(&ctrl->scan_work);
> +
>  	/*
>  	 * The dead states indicates the controller was not gracefully
>  	 * disconnected. In that case, we won't be able to flush any data while
> @@ -3489,7 +3492,6 @@ void nvme_stop_ctrl(struct nvme_ctrl *ctrl)
>  	nvme_mpath_stop(ctrl);
>  	nvme_stop_keep_alive(ctrl);
>  	flush_work(&ctrl->async_event_work);
> -	flush_work(&ctrl->scan_work);
>  	cancel_work_sync(&ctrl->fw_act_work);
>  	if (ctrl->ops->stop_ctrl)
>  		ctrl->ops->stop_ctrl(ctrl);
> -- 

[PATCH] nvme-core: flush namespace scanning work just before removing namespaces

[James Smart]

On 11/21/2018 3:17 PM, Sagi Grimberg wrote:
> nvme_stop_ctrl can be called also for reset flow and there is no need to
> flush the scan_work as namespaces are not being removed. This can cause
> deadlock in rdma, fc and loop drivers since nvme_stop_ctrl barriers
> before controller teardown (and specifically I/O cancellation of the
> scan_work itself) takes place, but the scan_work will be blocked anyways
> so there is no need to flush it.
>
> Instead, move scan_work flush to nvme_remove_namespaces() where it really
> needs to flush.
>
> Reported-by: Ming Lei <ming.lei at redhat.com>
> Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
> ---
> This is a stable candidate...
>
>   drivers/nvme/host/core.c | 4 +++-
>   1 file changed, 3 insertions(+), 1 deletion(-)
>
Looks good to me.

Reviewed by:? James Smart? <jsmart2021 at gmail.com>

-- james

[PATCH] nvme-core: flush namespace scanning work just before removing namespaces

[Christoph Hellwig]

Thanks,

applied to nvme-4.20.