[PATCH] nvme-multipath: fix multipath path scan

[PATCH] nvme-multipath: fix multipath path scan

[Kanchan Joshi]

commit e7d65803e2bb ("nvme-multipath: revalidate paths during rescan")
introduced NVME_NS_READY flag. nvme_path_is_disabled() declares path/ns
to be disabled if this flag is not found to be set.
This causes side-effect on namespace for which block-interface is not
fully up.

> nvme id-ns /dev/ng1n1
identify namespace: Resource temporarily unavailable

Above issue occurs because nvme_find_path() fails as it does not find
the NVME_NS_READY set.

       ns = nvme_find_path(head);
       if (!ns)
                goto out_unlock;

Go back to old behaviour by moving up setting of NVME_NS_READY.

After this patch:
> nvme id-ns /dev/ng1n1
NVME Identify Namespace 1:
nsze    : 0x3cf00000
ncap    : 0x3cf00000
nuse    : 0
...........

Fixes: e7d65803e2bb ("nvme-multipath: revalidate paths during rescan")
Signed-off-by: Kanchan Joshi <joshi.k@samsung.com>
---
 drivers/nvme/host/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 961a5f8a44d2..71c1970f029d 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -1914,6 +1914,7 @@ static int nvme_update_ns_info(struct nvme_ns *ns, struct nvme_id_ns *id)
 	blk_mq_freeze_queue(ns->disk->queue);
 	ns->lba_shift = id->lbaf[lbaf].ds;
 	nvme_set_queue_limits(ns->ctrl, ns->queue);
+	set_bit(NVME_NS_READY, &ns->flags);
 
 	ret = nvme_configure_metadata(ns, id);
 	if (ret)
@@ -1927,7 +1928,6 @@ static int nvme_update_ns_info(struct nvme_ns *ns, struct nvme_id_ns *id)
 			goto out_unfreeze;
 	}
 
-	set_bit(NVME_NS_READY, &ns->flags);
 	blk_mq_unfreeze_queue(ns->disk->queue);
 
 	if (blk_queue_is_zoned(ns->queue)) {
-- 
2.25.1

Re: [PATCH] nvme-multipath: fix multipath path scan

[Christoph Hellwig]

On Wed, Feb 16, 2022 at 11:43:05AM +0530, Kanchan Joshi wrote:
> commit e7d65803e2bb ("nvme-multipath: revalidate paths during rescan")
> introduced NVME_NS_READY flag. nvme_path_is_disabled() declares path/ns
> to be disabled if this flag is not found to be set.
> This causes side-effect on namespace for which block-interface is not
> fully up.

Hmm. This means for non-bdev namespaces we error out somewhere in
nvme_update_ns_info.  Where do we fail here, nvme_configure_metadata?

Re: [PATCH] nvme-multipath: fix multipath path scan

[Kanchan Joshi]

On Wed, Feb 16, 2022 at 1:41 PM Christoph Hellwig <hch@lst.de> wrote:
>
> On Wed, Feb 16, 2022 at 11:43:05AM +0530, Kanchan Joshi wrote:
> > commit e7d65803e2bb ("nvme-multipath: revalidate paths during rescan")
> > introduced NVME_NS_READY flag. nvme_path_is_disabled() declares path/ns
> > to be disabled if this flag is not found to be set.
> > This causes side-effect on namespace for which block-interface is not
> > fully up.
>
> Hmm. This means for non-bdev namespaces we error out somewhere in
> nvme_update_ns_info.  Where do we fail here, nvme_configure_metadata?

In nvme_update_zone_info, but nvme_configure_metadata is also possible.
Moved setting the flag before both these.


-- 
Kanchan

Re: [PATCH] nvme-multipath: fix multipath path scan

[Daniel Wagner]

On Wed, Feb 16, 2022 at 06:18:06PM +0530, Kanchan Joshi wrote:
> On Wed, Feb 16, 2022 at 1:41 PM Christoph Hellwig <hch@lst.de> wrote:
> >
> > On Wed, Feb 16, 2022 at 11:43:05AM +0530, Kanchan Joshi wrote:
> > > commit e7d65803e2bb ("nvme-multipath: revalidate paths during rescan")
> > > introduced NVME_NS_READY flag. nvme_path_is_disabled() declares path/ns
> > > to be disabled if this flag is not found to be set.
> > > This causes side-effect on namespace for which block-interface is not
> > > fully up.
> >
> > Hmm. This means for non-bdev namespaces we error out somewhere in
> > nvme_update_ns_info.  Where do we fail here, nvme_configure_metadata?
> 
> In nvme_update_zone_info, but nvme_configure_metadata is also possible.
> Moved setting the flag before both these.

But the NS is not ready at this point. nvme_update_disk_info() needs run
before the NS can be marked as ready. Commit message e7d65803e2bb
("nvme-multipath: revalidate paths during rescan") explains why.

Re: [PATCH] nvme-multipath: fix multipath path scan

[Christoph Hellwig]

On Wed, Feb 16, 2022 at 01:57:31PM +0100, Daniel Wagner wrote:
> > In nvme_update_zone_info, but nvme_configure_metadata is also possible.
> > Moved setting the flag before both these.
> 
> But the NS is not ready at this point. nvme_update_disk_info() needs run
> before the NS can be marked as ready. Commit message e7d65803e2bb
> ("nvme-multipath: revalidate paths during rescan") explains why.

Yeah.  So the only error return from nvme_configure_metadata is a can not
happen case which doesn't really need the error return.  So we only
need to check nvme_update_zone_info and can set the ns life for the
-ENODEV case there.  Something like the patch below:


diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 79005ea1a33e3..7814df10b5199 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -1723,7 +1723,7 @@ static int nvme_setup_streams_ns(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 	return 0;
 }
 
-static int nvme_configure_metadata(struct nvme_ns *ns, struct nvme_id_ns *id)
+static void nvme_configure_metadata(struct nvme_ns *ns, struct nvme_id_ns *id)
 {
 	struct nvme_ctrl *ctrl = ns->ctrl;
 
@@ -1739,7 +1739,8 @@ static int nvme_configure_metadata(struct nvme_ns *ns, struct nvme_id_ns *id)
 
 	ns->features &= ~(NVME_NS_METADATA_SUPPORTED | NVME_NS_EXT_LBAS);
 	if (!ns->ms || !(ctrl->ops->flags & NVME_F_METADATA_SUPPORTED))
-		return 0;
+		return;
+
 	if (ctrl->ops->flags & NVME_F_FABRICS) {
 		/*
 		 * The NVMe over Fabrics specification only supports metadata as
@@ -1747,7 +1748,7 @@ static int nvme_configure_metadata(struct nvme_ns *ns, struct nvme_id_ns *id)
 		 * remap the separate metadata buffer from the block layer.
 		 */
 		if (WARN_ON_ONCE(!(id->flbas & NVME_NS_FLBAS_META_EXT)))
-			return -EINVAL;
+			return;
 
 		ns->features |= NVME_NS_EXT_LBAS;
 
@@ -1774,8 +1775,6 @@ static int nvme_configure_metadata(struct nvme_ns *ns, struct nvme_id_ns *id)
 		else
 			ns->features |= NVME_NS_METADATA_SUPPORTED;
 	}
-
-	return 0;
 }
 
 static void nvme_set_queue_limits(struct nvme_ctrl *ctrl,
@@ -1916,9 +1915,7 @@ static int nvme_update_ns_info(struct nvme_ns *ns, struct nvme_id_ns *id)
 	ns->lba_shift = id->lbaf[lbaf].ds;
 	nvme_set_queue_limits(ns->ctrl, ns->queue);
 
-	ret = nvme_configure_metadata(ns, id);
-	if (ret)
-		goto out_unfreeze;
+	nvme_configure_metadata(ns, id);
 	nvme_set_chunk_sectors(ns, id);
 	nvme_update_disk_info(ns->disk, ns, id);
 
@@ -1934,7 +1931,7 @@ static int nvme_update_ns_info(struct nvme_ns *ns, struct nvme_id_ns *id)
 	if (blk_queue_is_zoned(ns->queue)) {
 		ret = nvme_revalidate_zones(ns);
 		if (ret && !nvme_first_scan(ns->disk))
-			goto out;
+			return ret;
 	}
 
 	if (nvme_ns_head_multipath(ns->head)) {
@@ -1949,16 +1946,16 @@ static int nvme_update_ns_info(struct nvme_ns *ns, struct nvme_id_ns *id)
 	return 0;
 
 out_unfreeze:
-	blk_mq_unfreeze_queue(ns->disk->queue);
-out:
 	/*
 	 * If probing fails due an unsupported feature, hide the block device,
 	 * but still allow other access.
 	 */
 	if (ret == -ENODEV) {
 		ns->disk->flags |= GENHD_FL_HIDDEN;
+		set_bit(NVME_NS_READY, &ns->flags);
 		ret = 0;
 	}
+	blk_mq_unfreeze_queue(ns->disk->queue);
 	return ret;
 }
 

Re: [PATCH] nvme-multipath: fix multipath path scan

[Kanchan Joshi]

On Wed, Feb 16, 2022 at 6:50 PM Christoph Hellwig <hch@lst.de> wrote:
>
> On Wed, Feb 16, 2022 at 01:57:31PM +0100, Daniel Wagner wrote:
> > > In nvme_update_zone_info, but nvme_configure_metadata is also possible.
> > > Moved setting the flag before both these.
> >
> > But the NS is not ready at this point. nvme_update_disk_info() needs run
> > before the NS can be marked as ready. Commit message e7d65803e2bb
> > ("nvme-multipath: revalidate paths during rescan") explains why.
>
> Yeah.  So the only error return from nvme_configure_metadata is a can not
> happen case which doesn't really need the error return.  So we only
> need to check nvme_update_zone_info and can set the ns life for the
> -ENODEV case there.  Something like the patch below:

It looks good to me. Can add a tested-by too if you are planning to send it?

Thanks,
-- 
Kanchan

Re: [PATCH] nvme-multipath: fix multipath path scan

[Adam Manzanares]

On Wed, Feb 16, 2022 at 02:20:56PM +0100, Christoph Hellwig wrote:
> On Wed, Feb 16, 2022 at 01:57:31PM +0100, Daniel Wagner wrote:
> > > In nvme_update_zone_info, but nvme_configure_metadata is also possible.
> > > Moved setting the flag before both these.
> > 
> > But the NS is not ready at this point. nvme_update_disk_info() needs run
> > before the NS can be marked as ready. Commit message e7d65803e2bb
> > ("nvme-multipath: revalidate paths during rescan") explains why.
> 
> Yeah.  So the only error return from nvme_configure_metadata is a can not
> happen case which doesn't really need the error return.  So we only
> need to check nvme_update_zone_info and can set the ns life for the
> -ENODEV case there.  Something like the patch below:
> 
> 
> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> index 79005ea1a33e3..7814df10b5199 100644
> --- a/drivers/nvme/host/core.c
> +++ b/drivers/nvme/host/core.c
> @@ -1723,7 +1723,7 @@ static int nvme_setup_streams_ns(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
>  	return 0;
>  }
>  
> -static int nvme_configure_metadata(struct nvme_ns *ns, struct nvme_id_ns *id)
> +static void nvme_configure_metadata(struct nvme_ns *ns, struct nvme_id_ns *id)
>  {
>  	struct nvme_ctrl *ctrl = ns->ctrl;
>  
> @@ -1739,7 +1739,8 @@ static int nvme_configure_metadata(struct nvme_ns *ns, struct nvme_id_ns *id)
>  
>  	ns->features &= ~(NVME_NS_METADATA_SUPPORTED | NVME_NS_EXT_LBAS);
>  	if (!ns->ms || !(ctrl->ops->flags & NVME_F_METADATA_SUPPORTED))
> -		return 0;
> +		return;
> +
>  	if (ctrl->ops->flags & NVME_F_FABRICS) {
>  		/*
>  		 * The NVMe over Fabrics specification only supports metadata as
> @@ -1747,7 +1748,7 @@ static int nvme_configure_metadata(struct nvme_ns *ns, struct nvme_id_ns *id)
>  		 * remap the separate metadata buffer from the block layer.
>  		 */
>  		if (WARN_ON_ONCE(!(id->flbas & NVME_NS_FLBAS_META_EXT)))
> -			return -EINVAL;
> +			return;
>  
>  		ns->features |= NVME_NS_EXT_LBAS;
>  
> @@ -1774,8 +1775,6 @@ static int nvme_configure_metadata(struct nvme_ns *ns, struct nvme_id_ns *id)
>  		else
>  			ns->features |= NVME_NS_METADATA_SUPPORTED;
>  	}
> -
> -	return 0;
>  }
>  
>  static void nvme_set_queue_limits(struct nvme_ctrl *ctrl,
> @@ -1916,9 +1915,7 @@ static int nvme_update_ns_info(struct nvme_ns *ns, struct nvme_id_ns *id)
>  	ns->lba_shift = id->lbaf[lbaf].ds;
>  	nvme_set_queue_limits(ns->ctrl, ns->queue);
>  
> -	ret = nvme_configure_metadata(ns, id);
> -	if (ret)
> -		goto out_unfreeze;
> +	nvme_configure_metadata(ns, id);
>  	nvme_set_chunk_sectors(ns, id);
>  	nvme_update_disk_info(ns->disk, ns, id);
>  
> @@ -1934,7 +1931,7 @@ static int nvme_update_ns_info(struct nvme_ns *ns, struct nvme_id_ns *id)
>  	if (blk_queue_is_zoned(ns->queue)) {
>  		ret = nvme_revalidate_zones(ns);
>  		if (ret && !nvme_first_scan(ns->disk))
> -			goto out;
> +			return ret;
>  	}
>  
>  	if (nvme_ns_head_multipath(ns->head)) {
> @@ -1949,16 +1946,16 @@ static int nvme_update_ns_info(struct nvme_ns *ns, struct nvme_id_ns *id)
>  	return 0;
>  
>  out_unfreeze:
> -	blk_mq_unfreeze_queue(ns->disk->queue);
> -out:
>  	/*
>  	 * If probing fails due an unsupported feature, hide the block device,
>  	 * but still allow other access.
>  	 */
>  	if (ret == -ENODEV) {
>  		ns->disk->flags |= GENHD_FL_HIDDEN;
> +		set_bit(NVME_NS_READY, &ns->flags);
>  		ret = 0;
>  	}
> +	blk_mq_unfreeze_queue(ns->disk->queue);
>  	return ret;
>  }

LGTM

Reviewed-by: Adam Manzanares <a.manzanares@samsung.com>

>