[PATCH 0/5] scsi: Support to handle Intermittent errors

[PATCH 0/5] scsi: Support to handle Intermittent errors

[Muneendra]

This patch adds a support to prevent retries of all the pending/inflight
io's after an abort succeeds on a particular device when transport
connectivity to the device is encountering intermittent errors.

Intermittent connectivity is a condition that can be detected by transport
fabric notifications. A service can monitor the ELS notifications and
take action on all the outstanding io's of a scsi device at that instant.


This feature is intended to be used when the device is part of a multipath
environment. When the service detects the poor connectivity, the multipath
path can be placed in a marginal path group and ignored further io
operations.

After placing a path in the marginal path group,the daemon sets a bit in
scmd->state for all the outstanding io's on that particular device with
the new sysfs interface provided in this patch.This prevent retries of 
all the pending/inflight io's if an io hits a scsi timeout which inturn
issues an abort.On Abort succeeds on a marginal path the io will be
immediately retried on another active path.On abort fails then the things
escalates to existing target reset sg interface recovery process.

Below is the interface provided to abort the io
echo 1 >> /sys/class/fc_transport/targetX\:Y\:Z/noretries_abort

The patches were cut against  5.9/scsi-queue tree

Muneendra (5):
  scsi: Added a new macro in scsi_cmnd.h
  scsi: Clear state bit SCMD_NORETRIES_ABORT of scsi_cmd before start
    request
  scsi: No retries on abort success
  scsi: Added routine to set SCMD_NORETRIES_ABORT bit for outstanding io
    on scsi_dev
  scsi_transport_fc: Added a new sysfs attribute noretries_abort

 drivers/scsi/scsi_error.c        | 63 ++++++++++++++++++++++++++++++++++++++++
 drivers/scsi/scsi_lib.c          |  8 +++--
 drivers/scsi/scsi_priv.h         |  1 +
 drivers/scsi/scsi_transport_fc.c | 49 +++++++++++++++++++++++++++++--
 include/scsi/scsi_cmnd.h         |  3 ++
 5 files changed, 120 insertions(+), 4 deletions(-)

-- 
1.8.3.1

[PATCH 1/5] scsi: Added a new macro in scsi_cmnd.h

[Muneendra]

Added a new macro SCMD_NORETRIES_ABORT in scsi_cmnd.h

The SCMD_NORETRIES_ABORT macro  defines the third bit postion
in scmd->state

Signed-off-by: Muneendra <muneendra.kumar@broadcom.com>
---
 include/scsi/scsi_cmnd.h | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/include/scsi/scsi_cmnd.h b/include/scsi/scsi_cmnd.h
index e76bac4..e1883fe 100644
--- a/include/scsi/scsi_cmnd.h
+++ b/include/scsi/scsi_cmnd.h
@@ -64,6 +64,9 @@ struct scsi_pointer {
 /* for scmd->state */
 #define SCMD_STATE_COMPLETE	0
 #define SCMD_STATE_INFLIGHT	1
+#define SCMD_NORETRIES_ABORT	2 /* If this bit is set then there won't be any
+				   * retries of scmd on abort success
+				   */
 
 struct scsi_cmnd {
 	struct scsi_request req;
-- 
1.8.3.1

Re: [PATCH 1/5] scsi: Added a new macro in scsi_cmnd.h

[Hannes Reinecke]

On 8/5/20 4:50 AM, Muneendra wrote:
> Added a new macro SCMD_NORETRIES_ABORT in scsi_cmnd.h
> 
It's not a macro, it's a definition.

> The SCMD_NORETRIES_ABORT macro  defines the third bit postion
> in scmd->state
> 
> Signed-off-by: Muneendra <muneendra.kumar@broadcom.com>
> ---
>   include/scsi/scsi_cmnd.h | 3 +++
>   1 file changed, 3 insertions(+)
> 
> diff --git a/include/scsi/scsi_cmnd.h b/include/scsi/scsi_cmnd.h
> index e76bac4..e1883fe 100644
> --- a/include/scsi/scsi_cmnd.h
> +++ b/include/scsi/scsi_cmnd.h
> @@ -64,6 +64,9 @@ struct scsi_pointer {
>   /* for scmd->state */
>   #define SCMD_STATE_COMPLETE	0
>   #define SCMD_STATE_INFLIGHT	1
> +#define SCMD_NORETRIES_ABORT	2 /* If this bit is set then there won't be any
> +				   * retries of scmd on abort success
> +				   */
>   
>   struct scsi_cmnd {
>   	struct scsi_request req;
> 
Other than that:

Reviewed-by: Hannes Reinecke <hare@suse.de>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke            Teamlead Storage & Networking
hare@suse.de                               +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

[PATCH 2/5] scsi: Clear state bit SCMD_NORETRIES_ABORT of scsi_cmd before start request

[Muneendra]

Clearing the SCMD_NORETRIES_ABORT bit in state flag of scsi_cmd if the
block layer didn't complete the request due to a timeout injection so that
the timeout handler will see it needs to escalate its own error recovery.
Also clearing the SCMD_NORETRIES_ABORT bit in state flag before
blk_mq_start_request.

Signed-off-by: Muneendra <muneendra.kumar@broadcom.com>
---
 drivers/scsi/scsi_lib.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index 27b52fc..3da6402 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -1594,12 +1594,15 @@ static void scsi_mq_done(struct scsi_cmnd *cmd)
 
 	/*
 	 * If the block layer didn't complete the request due to a timeout
-	 * injection, scsi must clear its internal completed state so that the
+	 * injection, scsi must clear its internal completed state and
+	 * SCMD_NORETRIES_ABORT bit in state field  so that the
 	 * timeout handler will see it needs to escalate its own error
 	 * recovery.
 	 */
-	if (unlikely(!blk_mq_complete_request(cmd->request)))
+	if (unlikely(!blk_mq_complete_request(cmd->request))) {
 		clear_bit(SCMD_STATE_COMPLETE, &cmd->state);
+		clear_bit(SCMD_NORETRIES_ABORT, &cmd->state);
+	}
 }
 
 static void scsi_mq_put_budget(struct blk_mq_hw_ctx *hctx)
@@ -1652,6 +1655,7 @@ static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx,
 		req->rq_flags |= RQF_DONTPREP;
 	} else {
 		clear_bit(SCMD_STATE_COMPLETE, &cmd->state);
+		clear_bit(SCMD_NORETRIES_ABORT, &cmd->state);
 		blk_mq_start_request(req);
 	}
 
-- 
1.8.3.1

Re: [PATCH 2/5] scsi: Clear state bit SCMD_NORETRIES_ABORT of scsi_cmd before start request

[Hannes Reinecke]

On 8/5/20 4:50 AM, Muneendra wrote:
> Clearing the SCMD_NORETRIES_ABORT bit in state flag of scsi_cmd if the
> block layer didn't complete the request due to a timeout injection so that
> the timeout handler will see it needs to escalate its own error recovery.
> Also clearing the SCMD_NORETRIES_ABORT bit in state flag before
> blk_mq_start_request.
> 
> Signed-off-by: Muneendra <muneendra.kumar@broadcom.com>
> ---
>   drivers/scsi/scsi_lib.c | 8 ++++++--
>   1 file changed, 6 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
> index 27b52fc..3da6402 100644
> --- a/drivers/scsi/scsi_lib.c
> +++ b/drivers/scsi/scsi_lib.c
> @@ -1594,12 +1594,15 @@ static void scsi_mq_done(struct scsi_cmnd *cmd)
>   
>   	/*
>   	 * If the block layer didn't complete the request due to a timeout
> -	 * injection, scsi must clear its internal completed state so that the
> +	 * injection, scsi must clear its internal completed state and
> +	 * SCMD_NORETRIES_ABORT bit in state field  so that the
>   	 * timeout handler will see it needs to escalate its own error
>   	 * recovery.
>   	 */
> -	if (unlikely(!blk_mq_complete_request(cmd->request)))
> +	if (unlikely(!blk_mq_complete_request(cmd->request))) {
>   		clear_bit(SCMD_STATE_COMPLETE, &cmd->state);
> +		clear_bit(SCMD_NORETRIES_ABORT, &cmd->state);
> +	}
>   }
>   
>   static void scsi_mq_put_budget(struct blk_mq_hw_ctx *hctx)
> @@ -1652,6 +1655,7 @@ static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx,
>   		req->rq_flags |= RQF_DONTPREP;
>   	} else {
>   		clear_bit(SCMD_STATE_COMPLETE, &cmd->state);
> +		clear_bit(SCMD_NORETRIES_ABORT, &cmd->state);
>   		blk_mq_start_request(req);
>   	}
>   
> 
Reviewed-by: Hannes Reinecke <hare@suse.de>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke            Teamlead Storage & Networking
hare@suse.de                               +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

[PATCH 3/5] scsi: No retries on abort success

[Muneendra]

Made an additional check in scsi_noretry_cmd to verify whether user has
decided not to do retries on abort(issued on scsi timeouts) success  by
checking the SCMD_NORETRIES_ABORT bit

If SCMD_NORETRIES_ABORT bit is set we are making sure there won't be any
retries done on the same path and also setting the host byte as
DID_TRANSPORT_FAILFAST so that the error can be propogated as recoverable
transport error to the blk layers.

Signed-off-by: Muneendra <muneendra.kumar@broadcom.com>
---
 drivers/scsi/scsi_error.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/drivers/scsi/scsi_error.c b/drivers/scsi/scsi_error.c
index 927b1e6..3222496 100644
--- a/drivers/scsi/scsi_error.c
+++ b/drivers/scsi/scsi_error.c
@@ -1749,6 +1749,16 @@ int scsi_noretry_cmd(struct scsi_cmnd *scmd)
 
 check_type:
 	/*
+	 * Check whether caller has decided not to do retries on
+	 * abort success by setting the SCMD_NORETRIES_ABORT bit
+	 */
+	if ((test_bit(SCMD_NORETRIES_ABORT, &scmd->state)) &&
+		(scmd->request->cmd_flags & REQ_FAILFAST_TRANSPORT)) {
+		set_host_byte(scmd, DID_TRANSPORT_FAILFAST);
+		return 1;
+	}
+
+	/*
 	 * assume caller has checked sense and determined
 	 * the check condition was retryable.
 	 */
-- 
1.8.3.1

Re: [PATCH 3/5] scsi: No retries on abort success

[Hannes Reinecke]

On 8/5/20 4:51 AM, Muneendra wrote:
> Made an additional check in scsi_noretry_cmd to verify whether user has
> decided not to do retries on abort(issued on scsi timeouts) success  by
> checking the SCMD_NORETRIES_ABORT bit
> 
> If SCMD_NORETRIES_ABORT bit is set we are making sure there won't be any
> retries done on the same path and also setting the host byte as
> DID_TRANSPORT_FAILFAST so that the error can be propogated as recoverable
> transport error to the blk layers.
> 
> Signed-off-by: Muneendra <muneendra.kumar@broadcom.com>
> ---
>   drivers/scsi/scsi_error.c | 10 ++++++++++
>   1 file changed, 10 insertions(+)
> 
> diff --git a/drivers/scsi/scsi_error.c b/drivers/scsi/scsi_error.c
> index 927b1e6..3222496 100644
> --- a/drivers/scsi/scsi_error.c
> +++ b/drivers/scsi/scsi_error.c
> @@ -1749,6 +1749,16 @@ int scsi_noretry_cmd(struct scsi_cmnd *scmd)
>   
>   check_type:
>   	/*
> +	 * Check whether caller has decided not to do retries on
> +	 * abort success by setting the SCMD_NORETRIES_ABORT bit
> +	 */
> +	if ((test_bit(SCMD_NORETRIES_ABORT, &scmd->state)) &&
> +		(scmd->request->cmd_flags & REQ_FAILFAST_TRANSPORT)) {
> +		set_host_byte(scmd, DID_TRANSPORT_FAILFAST);
> +		return 1;
> +	}
> +
> +	/*
>   	 * assume caller has checked sense and determined
>   	 * the check condition was retryable.
>   	 */
> 
_Actually_ DID_TRANSPORT_FAILFAST is just for transport aborted 
commands, so maybe we should use a different error code (or add a new one).
But other than that:

Reviewed-by: Hannes Reinecke <hare@suse.de>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke            Teamlead Storage & Networking
hare@suse.de                               +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

[PATCH 4/5] scsi: Added routine to set SCMD_NORETRIES_ABORT bit for outstanding io on scsi_dev

[Muneendra]

Added a new routine scsi_set_noretries_abort_io_device()to set
SCMD_NORETRIES_ABORT for all the inflight/pending IO's on a particular
scsi device at that particular instant.

Export the symbol so the routine can be called by scsi_transport_fc.c

Added new function declaration scsi_set_noretries_abort_io_device in
scsi_priv.h

Signed-off-by: Muneendra <muneendra.kumar@broadcom.com>
---
 drivers/scsi/scsi_error.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++
 drivers/scsi/scsi_priv.h  |  1 +
 2 files changed, 54 insertions(+)

diff --git a/drivers/scsi/scsi_error.c b/drivers/scsi/scsi_error.c
index 3222496..938d770 100644
--- a/drivers/scsi/scsi_error.c
+++ b/drivers/scsi/scsi_error.c
@@ -271,6 +271,59 @@ void scsi_eh_scmd_add(struct scsi_cmnd *scmd)
 	call_rcu(&scmd->rcu, scsi_eh_inc_host_failed);
 }
 
+static bool
+scsi_set_noretries_abort_io(struct request *rq, void *priv, bool reserved)
+{
+	struct scsi_cmnd *scmd = blk_mq_rq_to_pdu(rq);
+	struct scsi_device *sdev = scmd->device;
+
+	/* only set SCMD_NORETRIES_ABORT on ios on a specific sdev */
+	if (sdev != priv)
+		return true;
+	/* we don't want this command reissued on abort success
+	 * so set SCMD_NORETRIES_ABORT bit to ensure it
+	 * won't get reissued
+	 */
+	if (READ_ONCE(rq->state) == MQ_RQ_IN_FLIGHT)
+		set_bit(SCMD_NORETRIES_ABORT, &scmd->state);
+	return true;
+}
+
+static int
+__scsi_set_noretries_abort_io_device(struct scsi_device *sdev)
+{
+
+	if (sdev->sdev_state != SDEV_RUNNING)
+		return -EINVAL;
+
+	if (blk_queue_init_done(sdev->request_queue)) {
+
+		blk_mq_quiesce_queue(sdev->request_queue);
+		blk_mq_tagset_busy_iter(&sdev->host->tag_set,
+				scsi_set_noretries_abort_io, sdev);
+		blk_mq_unquiesce_queue(sdev->request_queue);
+	}
+	return 0;
+}
+
+/*
+ * scsi_set_noretries_abort_io_device - set the SCMD_NORETRIES_ABORT
+ * bit for all the pending io's on a device
+ * @sdev:	scsi_device
+ */
+int
+scsi_set_noretries_abort_io_device(struct scsi_device *sdev)
+{
+	struct Scsi_Host *shost = sdev->host;
+	int ret  = -EINVAL;
+
+	mutex_lock(&shost->scan_mutex);
+	ret = __scsi_set_noretries_abort_io_device(sdev);
+	mutex_unlock(&shost->scan_mutex);
+	return ret;
+}
+EXPORT_SYMBOL(scsi_set_noretries_abort_io_device);
+
 /**
  * scsi_times_out - Timeout function for normal scsi commands.
  * @req:	request that is timing out.
diff --git a/drivers/scsi/scsi_priv.h b/drivers/scsi/scsi_priv.h
index d12ada0..1bbffd3 100644
--- a/drivers/scsi/scsi_priv.h
+++ b/drivers/scsi/scsi_priv.h
@@ -81,6 +81,7 @@ void scsi_eh_ready_devs(struct Scsi_Host *shost,
 int scsi_eh_get_sense(struct list_head *work_q,
 		      struct list_head *done_q);
 int scsi_noretry_cmd(struct scsi_cmnd *scmd);
+extern int scsi_set_noretries_abort_io_device(struct scsi_device *sdev);
 
 /* scsi_lib.c */
 extern int scsi_maybe_unblock_host(struct scsi_device *sdev);
-- 
1.8.3.1

Re: [PATCH 4/5] scsi: Added routine to set SCMD_NORETRIES_ABORT bit for outstanding io on scsi_dev

[Hannes Reinecke]

On 8/5/20 4:51 AM, Muneendra wrote:
> Added a new routine scsi_set_noretries_abort_io_device()to set
> SCMD_NORETRIES_ABORT for all the inflight/pending IO's on a particular
> scsi device at that particular instant.
> 
> Export the symbol so the routine can be called by scsi_transport_fc.c
> 
> Added new function declaration scsi_set_noretries_abort_io_device in
> scsi_priv.h
> 
> Signed-off-by: Muneendra <muneendra.kumar@broadcom.com>
> ---
>   drivers/scsi/scsi_error.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++
>   drivers/scsi/scsi_priv.h  |  1 +
>   2 files changed, 54 insertions(+)
> 
> diff --git a/drivers/scsi/scsi_error.c b/drivers/scsi/scsi_error.c
> index 3222496..938d770 100644
> --- a/drivers/scsi/scsi_error.c
> +++ b/drivers/scsi/scsi_error.c
> @@ -271,6 +271,59 @@ void scsi_eh_scmd_add(struct scsi_cmnd *scmd)
>   	call_rcu(&scmd->rcu, scsi_eh_inc_host_failed);
>   }
>   
> +static bool
> +scsi_set_noretries_abort_io(struct request *rq, void *priv, bool reserved)
> +{
> +	struct scsi_cmnd *scmd = blk_mq_rq_to_pdu(rq);
> +	struct scsi_device *sdev = scmd->device;
> +
> +	/* only set SCMD_NORETRIES_ABORT on ios on a specific sdev */
> +	if (sdev != priv)
> +		return true;
> +	/* we don't want this command reissued on abort success
> +	 * so set SCMD_NORETRIES_ABORT bit to ensure it
> +	 * won't get reissued
> +	 */
> +	if (READ_ONCE(rq->state) == MQ_RQ_IN_FLIGHT)
> +		set_bit(SCMD_NORETRIES_ABORT, &scmd->state);
> +	return true;
> +}
> +
> +static int
> +__scsi_set_noretries_abort_io_device(struct scsi_device *sdev)
> +{
> +
> +	if (sdev->sdev_state != SDEV_RUNNING)
> +		return -EINVAL;
> +
> +	if (blk_queue_init_done(sdev->request_queue)) {
> +
> +		blk_mq_quiesce_queue(sdev->request_queue);
> +		blk_mq_tagset_busy_iter(&sdev->host->tag_set,
> +				scsi_set_noretries_abort_io, sdev);
> +		blk_mq_unquiesce_queue(sdev->request_queue);
> +	}
> +	return 0;
> +}
> +
> +/*
> + * scsi_set_noretries_abort_io_device - set the SCMD_NORETRIES_ABORT
> + * bit for all the pending io's on a device
> + * @sdev:	scsi_device
> + */
> +int
> +scsi_set_noretries_abort_io_device(struct scsi_device *sdev)
> +{
> +	struct Scsi_Host *shost = sdev->host;
> +	int ret  = -EINVAL;
> +
> +	mutex_lock(&shost->scan_mutex);
> +	ret = __scsi_set_noretries_abort_io_device(sdev);
> +	mutex_unlock(&shost->scan_mutex);
> +	return ret;
> +}
> +EXPORT_SYMBOL(scsi_set_noretries_abort_io_device);
> +
>   /**
>    * scsi_times_out - Timeout function for normal scsi commands.
>    * @req:	request that is timing out.
> diff --git a/drivers/scsi/scsi_priv.h b/drivers/scsi/scsi_priv.h
> index d12ada0..1bbffd3 100644
> --- a/drivers/scsi/scsi_priv.h
> +++ b/drivers/scsi/scsi_priv.h
> @@ -81,6 +81,7 @@ void scsi_eh_ready_devs(struct Scsi_Host *shost,
>   int scsi_eh_get_sense(struct list_head *work_q,
>   		      struct list_head *done_q);
>   int scsi_noretry_cmd(struct scsi_cmnd *scmd);
> +extern int scsi_set_noretries_abort_io_device(struct scsi_device *sdev);
>   
>   /* scsi_lib.c */
>   extern int scsi_maybe_unblock_host(struct scsi_device *sdev);
> 
Reviewed-by: Hannes Reinecke <hare@suse.de>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke            Teamlead Storage & Networking
hare@suse.de                               +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

[PATCH 5/5] scsi_transport_fc: Added a new sysfs attribute noretries_abort

[Muneendra]

Added a new sysfs attribute noretries_abort under fc_transport/target*/

This interface will set SCMD_NORETRIES_ABORT bit in scmd->state for all
the pending io's on the scsi device associated with target port.

Below is the interface provided to abort the io
echo 1 >> /sys/class/fc_transport/targetX\:Y\:Z/noretries_abort

Signed-off-by: Muneendra <muneendra.kumar@broadcom.com>
---
 drivers/scsi/scsi_transport_fc.c | 49 ++++++++++++++++++++++++++++++++++++++--
 1 file changed, 47 insertions(+), 2 deletions(-)

diff --git a/drivers/scsi/scsi_transport_fc.c b/drivers/scsi/scsi_transport_fc.c
index 2732fa6..f7b00ae 100644
--- a/drivers/scsi/scsi_transport_fc.c
+++ b/drivers/scsi/scsi_transport_fc.c
@@ -305,7 +305,7 @@ struct device_attribute device_attr_##_prefix##_##_name = 	\
  * Attribute counts pre object type...
  * Increase these values if you add attributes
  */
-#define FC_STARGET_NUM_ATTRS 	3
+#define FC_STARGET_NUM_ATTRS	4
 #define FC_RPORT_NUM_ATTRS	10
 #define FC_VPORT_NUM_ATTRS	9
 #define FC_HOST_NUM_ATTRS	29
@@ -994,6 +994,44 @@ static FC_DEVICE_ATTR(rport, fast_io_fail_tmo, S_IRUGO | S_IWUSR,
 /*
  * FC SCSI Target Attribute Management
  */
+static void scsi_target_set_noretries_abort(struct scsi_target *starget)
+{
+	struct scsi_device *sdev, *tmp;
+	struct Scsi_Host *shost = dev_to_shost(starget->dev.parent);
+	unsigned long flags;
+
+	spin_lock_irqsave(shost->host_lock, flags);
+	list_for_each_entry_safe(sdev, tmp, &starget->devices, same_target_siblings) {
+		if (sdev->sdev_state == SDEV_DEL)
+			continue;
+		if (scsi_device_get(sdev))
+			continue;
+
+		spin_unlock_irqrestore(shost->host_lock, flags);
+		scsi_set_noretries_abort_io_device(sdev);
+		spin_lock_irqsave(shost->host_lock, flags);
+		scsi_device_put(sdev);
+	}
+	spin_unlock_irqrestore(shost->host_lock, flags);
+}
+
+/*
+ * Sets  no retries on abort in scmd->state for all
+ * outstanding io of all the scsi_devs
+ * write 1 to set the bit for all outstanding io's
+ */
+static ssize_t fc_target_set_noretries_abort(struct device *dev,
+						struct device_attribute *attr,
+						const char *buf, size_t count)
+{
+	struct scsi_target *starget = transport_class_to_starget(dev);
+
+	scsi_target_set_noretries_abort(starget);
+	return count;
+}
+
+static FC_DEVICE_ATTR(starget, noretries_abort, 0200,
+		NULL, fc_target_set_noretries_abort);
 
 /*
  * Note: in the target show function we recognize when the remote
@@ -1036,6 +1074,13 @@ static FC_DEVICE_ATTR(starget, field, S_IRUGO,			\
 	if (i->f->show_starget_##field)					\
 		count++
 
+#define SETUP_PRIVATE_STARGET_ATTRIBUTE_RW(field)			\
+do {									\
+	i->private_starget_attrs[count] = device_attr_starget_##field; \
+	i->starget_attrs[count] = &i->private_starget_attrs[count];	\
+	count++;							\
+} while (0)
+
 #define SETUP_STARGET_ATTRIBUTE_RW(field)				\
 	i->private_starget_attrs[count] = device_attr_starget_##field; \
 	if (!i->f->set_starget_##field) {				\
@@ -2197,7 +2242,7 @@ struct scsi_transport_template *
 	SETUP_STARGET_ATTRIBUTE_RD(node_name);
 	SETUP_STARGET_ATTRIBUTE_RD(port_name);
 	SETUP_STARGET_ATTRIBUTE_RD(port_id);
-
+	SETUP_PRIVATE_STARGET_ATTRIBUTE_RW(noretries_abort);
 	BUG_ON(count > FC_STARGET_NUM_ATTRS);
 
 	i->starget_attrs[count] = NULL;
-- 
1.8.3.1

Re: [PATCH 5/5] scsi_transport_fc: Added a new sysfs attribute noretries_abort

[Hannes Reinecke]

On 8/5/20 4:51 AM, Muneendra wrote:
> Added a new sysfs attribute noretries_abort under fc_transport/target*/
> 
> This interface will set SCMD_NORETRIES_ABORT bit in scmd->state for all
> the pending io's on the scsi device associated with target port.
> 
> Below is the interface provided to abort the io
> echo 1 >> /sys/class/fc_transport/targetX\:Y\:Z/noretries_abort
> 
> Signed-off-by: Muneendra <muneendra.kumar@broadcom.com>
> ---
>   drivers/scsi/scsi_transport_fc.c | 49 ++++++++++++++++++++++++++++++++++++++--
>   1 file changed, 47 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/scsi/scsi_transport_fc.c b/drivers/scsi/scsi_transport_fc.c
> index 2732fa6..f7b00ae 100644
> --- a/drivers/scsi/scsi_transport_fc.c
> +++ b/drivers/scsi/scsi_transport_fc.c
> @@ -305,7 +305,7 @@ struct device_attribute device_attr_##_prefix##_##_name = 	\
>    * Attribute counts pre object type...
>    * Increase these values if you add attributes
>    */
> -#define FC_STARGET_NUM_ATTRS 	3
> +#define FC_STARGET_NUM_ATTRS	4
>   #define FC_RPORT_NUM_ATTRS	10
>   #define FC_VPORT_NUM_ATTRS	9
>   #define FC_HOST_NUM_ATTRS	29
> @@ -994,6 +994,44 @@ static FC_DEVICE_ATTR(rport, fast_io_fail_tmo, S_IRUGO | S_IWUSR,
>   /*
>    * FC SCSI Target Attribute Management
>    */
> +static void scsi_target_set_noretries_abort(struct scsi_target *starget)
> +{
> +	struct scsi_device *sdev, *tmp;
> +	struct Scsi_Host *shost = dev_to_shost(starget->dev.parent);
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(shost->host_lock, flags);
> +	list_for_each_entry_safe(sdev, tmp, &starget->devices, same_target_siblings) {
> +		if (sdev->sdev_state == SDEV_DEL)
> +			continue;
> +		if (scsi_device_get(sdev))
> +			continue;
> +
> +		spin_unlock_irqrestore(shost->host_lock, flags);
> +		scsi_set_noretries_abort_io_device(sdev);
> +		spin_lock_irqsave(shost->host_lock, flags);
> +		scsi_device_put(sdev);
> +	}
> +	spin_unlock_irqrestore(shost->host_lock, flags);
> +}
> +
> +/*
> + * Sets  no retries on abort in scmd->state for all
> + * outstanding io of all the scsi_devs
> + * write 1 to set the bit for all outstanding io's
> + */
> +static ssize_t fc_target_set_noretries_abort(struct device *dev,
> +						struct device_attribute *attr,
> +						const char *buf, size_t count)
> +{
> +	struct scsi_target *starget = transport_class_to_starget(dev);
> +
> +	scsi_target_set_noretries_abort(starget);
> +	return count;
> +}
> +
> +static FC_DEVICE_ATTR(starget, noretries_abort, 0200,
> +		NULL, fc_target_set_noretries_abort);
>   
>   /*
>    * Note: in the target show function we recognize when the remote
> @@ -1036,6 +1074,13 @@ static FC_DEVICE_ATTR(starget, field, S_IRUGO,			\
>   	if (i->f->show_starget_##field)					\
>   		count++
>   
> +#define SETUP_PRIVATE_STARGET_ATTRIBUTE_RW(field)			\
> +do {									\
> +	i->private_starget_attrs[count] = device_attr_starget_##field; \
> +	i->starget_attrs[count] = &i->private_starget_attrs[count];	\
> +	count++;							\
> +} while (0)
> +
>   #define SETUP_STARGET_ATTRIBUTE_RW(field)				\
>   	i->private_starget_attrs[count] = device_attr_starget_##field; \
>   	if (!i->f->set_starget_##field) {				\
> @@ -2197,7 +2242,7 @@ struct scsi_transport_template *
>   	SETUP_STARGET_ATTRIBUTE_RD(node_name);
>   	SETUP_STARGET_ATTRIBUTE_RD(port_name);
>   	SETUP_STARGET_ATTRIBUTE_RD(port_id);
> -
> +	SETUP_PRIVATE_STARGET_ATTRIBUTE_RW(noretries_abort);
>   	BUG_ON(count > FC_STARGET_NUM_ATTRS);
>   
>   	i->starget_attrs[count] = NULL;
> 
Hmm. Wouldn't it make more sense to introduce a new port state 
'marginal' for this? We might want/need to introduce additional error 
recovery mechanisms here, so having a new state might be easier in the 
long run ...

Additionally, from my understanding the FPIN events will be generated 
with a certain frequency. So we could model the new 'marginal' state 
similar to the dev_loss_tmo mechanism; start a timer whenever the 
'marginal' state is being set, and clear the state back to 'running' if 
the state hasn't been refreshed within that timeframe.
That would give us an automatic state reset back to running, and quite 
easy to implement from userland.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke            Teamlead Storage & Networking
hare@suse.de                               +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

RE: [PATCH 5/5] scsi_transport_fc: Added a new sysfs attribute noretries_abort

[Muneendra Kumar M]

Hi Hannes,

>
>Hmm. Wouldn't it make more sense to introduce a new port state 'marginal'
>for this? We might >want/need to introduce additional error recovery
>mechanisms here, so having a new state >might be easier in the long run ...

>Additionally, from my understanding the FPIN events will be generated with
>a certain >frequency. So we could model the new 'marginal' state similar to
>the dev_loss_tmo >mechanism; start a timer whenever the 'marginal' state is
>being set, and clear the state back to >'running' if the state hasn't been
>refreshed within that timeframe.
>That would give us an automatic state reset back to running, and quite easy
>to implement from >userland.

Thanks for the review.
I have a small doubt.
When the port state moves from marginal to running state does it mean we
expect a traffic from the path ?

Regards,
Muneendra.

Re: [PATCH 5/5] scsi_transport_fc: Added a new sysfs attribute noretries_abort

[Hannes Reinecke]

On 8/11/20 8:01 AM, Muneendra Kumar M wrote:
> Hi Hannes,
> 
>>
>> Hmm. Wouldn't it make more sense to introduce a new port state 'marginal'
>> for this? We might >want/need to introduce additional error recovery
>> mechanisms here, so having a new state >might be easier in the long run ...
> 
>> Additionally, from my understanding the FPIN events will be generated with
>> a certain >frequency. So we could model the new 'marginal' state similar to
>> the dev_loss_tmo >mechanism; start a timer whenever the 'marginal' state is
>> being set, and clear the state back to >'running' if the state hasn't been
>> refreshed within that timeframe.
>> That would give us an automatic state reset back to running, and quite easy
>> to implement from >userland.
> 
> Thanks for the review.
> I have a small doubt.
> When the port state moves from marginal to running state does it mean we
> expect a traffic from the path ?
> 
We don't expect traffic; rather we _allow_ traffic.
But moving to from marginal to running means that we didn't receive FPIN
events, and the path should be considered healthy again.
So from that perspective it should be back to normal operations.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		           Kernel Storage Architect
hare@suse.de			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer

RE: [PATCH 5/5] scsi_transport_fc: Added a new sysfs attribute noretries_abort

[Muneendra Kumar M]

 Hi Hannes,
>>
>> Hmm. Wouldn't it make more sense to introduce a new port state 'marginal'
>> for this? We might >want/need to introduce additional error recovery
>> mechanisms here, so having a new state >might be easier in the long run
>> ...
>
>> Additionally, from my understanding the FPIN events will be generated
>> with a certain >frequency. So we could model the new 'marginal' state
>> similar to the dev_loss_tmo >mechanism; start a timer whenever the
>> 'marginal' state is being set, and clear the state back to >'running'
>> if the state hasn't been refreshed within that timeframe.
>> That would give us an automatic state reset back to running, and
>> quite easy to implement from >userland.
>
> Thanks for the review.
> I have a small doubt.
> When the port state moves from marginal to running state does it mean
> we expect a traffic from the path ?
>
>We don't expect traffic; rather we _allow_ traffic.
>But moving to from marginal to running means that we didn't receive FPIN
>events, and the path should be considered healthy again.
>So from that perspective it should be back to normal operations.


But this could  apply only to FPIN-Congestion. Only in this case FPIN-CN
FPIN events will be generated  with a certain  frequency.
But for FPIN-Li this is not the case.
FPIN-LI is used to inform about marginal paths, which needs manual
intervention to recover.
And for FPIN-LI the path should be re-enabled on any link bounce
(portdisable followed by portenable) which would correlated to a cable/sfp
change.
For now, however, we are addressing FPIN-LI primarily.

Regards,
Muneendra.

Re: [PATCH 5/5] scsi_transport_fc: Added a new sysfs attribute noretries_abort

[Hannes Reinecke]

On 8/11/20 9:03 AM, Muneendra Kumar M wrote:
>  Hi Hannes,
>>>
>>> Hmm. Wouldn't it make more sense to introduce a new port state 'marginal'
>>> for this? We might >want/need to introduce additional error recovery
>>> mechanisms here, so having a new state >might be easier in the long run
>>> ...
>>
>>> Additionally, from my understanding the FPIN events will be generated
>>> with a certain >frequency. So we could model the new 'marginal' state
>>> similar to the dev_loss_tmo >mechanism; start a timer whenever the
>>> 'marginal' state is being set, and clear the state back to >'running'
>>> if the state hasn't been refreshed within that timeframe.
>>> That would give us an automatic state reset back to running, and
>>> quite easy to implement from >userland.
>>
>> Thanks for the review.
>> I have a small doubt.
>> When the port state moves from marginal to running state does it mean
>> we expect a traffic from the path ?
>>
>> We don't expect traffic; rather we _allow_ traffic.
>> But moving to from marginal to running means that we didn't receive FPIN
>> events, and the path should be considered healthy again.
>> So from that perspective it should be back to normal operations.
> 
> 
> But this could  apply only to FPIN-Congestion. Only in this case FPIN-CN
> FPIN events will be generated  with a certain  frequency.
> But for FPIN-Li this is not the case.
> FPIN-LI is used to inform about marginal paths, which needs manual
> intervention to recover.
> And for FPIN-LI the path should be re-enabled on any link bounce
> (portdisable followed by portenable) which would correlated to a cable/sfp
> change.
> For now, however, we are addressing FPIN-LI primarily.
> 
Ah. So that changes things slightly; I had hoped we can address things
systematically, but with link integrity issues we don't have any other
choice but to replace the cable (ie wait for user interaction).

But still I'm in favour of the 'marginal' state; that one could be set
manually (or by an FPIN LI event), and would need to be reset either
manually or by link reset.

And have the advantage of being easier to implement :-)

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		           Kernel Storage Architect
hare@suse.de			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer

RE: [PATCH 5/5] scsi_transport_fc: Added a new sysfs attribute noretries_abort

[Muneendra Kumar M]

Hi Hannes,


>>>
>>> Hmm. Wouldn't it make more sense to introduce a new port state
>>> 'marginal'
>>> for this? We might >want/need to introduce additional error recovery
>>> mechanisms here, so having a new state >might be easier in the long
>>> run ...
>Ah. So that changes things slightly; I had hoped we can address things
>systematically, but with link integrity issues we don't have any other
>choice but to replace the cable (ie wait for user interaction).

>But still I'm in favour of the 'marginal' state; that one could be set
>manually (or by an FPIN LI event), and would need to be reset either
>manually or by link reset.

>And have the advantage of being easier to implement :-)

Thanks for the review.
I will incorporate all your review comments and will add marginal state in
my next version.

Regards,
Muneendra.