SCSI scanning behavior

SCSI scanning behavior

[Brian King]

In one of our SAN test labs where there is some storage controller error injection going on,
I'm seeing some interesting behavior. We are getting into a scenario, when the target is coming
back where we are going through SCSI scan for it and the Report LUNs we are issuing to it times
out, so we fall back to a sequential LUN scan. When performing the sequential LUN scan, we 
end up adding a bunch of LUNs than we didn't previously see, 512 in fact. The target is reporting
PQ=1, PDT=0 for every LUN that doesn't exist. When Report LUNs *does work*, it doesn't report
these LUNs. 

In net, we end up with a different result if we do a sequential LUN scan compared to a report LUNs
scan.

Now, one could argue this is a defect in the SCSI target, since SPC says:

The REPORT LUNS parameter data should be returned even though the device server is not ready for other
commands. The report of the logical unit inventory should be available without incurring any media access
delays. If the device server is not ready with the logical unit inventory or if the inventory list is null for the
requesting I_T nexus and the SELECT REPORT field set to 02h, then the device server shall provide a default
logical unit inventory that contains at least LUN 0 or the REPORT LUNS well known logical unit (see 8.2). A
non-empty peripheral device logical unit inventory that does not contain either LUN 0 or the REPORT LUNS
well known logical unit is valid.

However, I'm still left wondering why we are adding PQ=1, PDT=0 devices in the sequential LUN scan at all.
Are there media changer devices out there that we've seen respond like this? Even so, does it make sense
to add PQ=1, PDT=0 LUNs for LUN > 0?

Thanks,

Brian

-- 
Brian King
Power Linux I/O
IBM Linux Technology Center

Re: SCSI scanning behavior

[Hannes Reinecke]

On 08/21/2015 12:11 AM, Brian King wrote:
> In one of our SAN test labs where there is some storage controller error injection going on,
> I'm seeing some interesting behavior. We are getting into a scenario, when the target is coming
> back where we are going through SCSI scan for it and the Report LUNs we are issuing to it times
> out, so we fall back to a sequential LUN scan. When performing the sequential LUN scan, we 
> end up adding a bunch of LUNs than we didn't previously see, 512 in fact. The target is reporting
> PQ=1, PDT=0 for every LUN that doesn't exist. When Report LUNs *does work*, it doesn't report
> these LUNs. 
> 
> In net, we end up with a different result if we do a sequential LUN scan compared to a report LUNs
> scan.
> 
> Now, one could argue this is a defect in the SCSI target, since SPC says:
> 
> The REPORT LUNS parameter data should be returned even though the device server is not ready for other
> commands. The report of the logical unit inventory should be available without incurring any media access
> delays. If the device server is not ready with the logical unit inventory or if the inventory list is null for the
> requesting I_T nexus and the SELECT REPORT field set to 02h, then the device server shall provide a default
> logical unit inventory that contains at least LUN 0 or the REPORT LUNS well known logical unit (see 8.2). A
> non-empty peripheral device logical unit inventory that does not contain either LUN 0 or the REPORT LUNS
> well known logical unit is valid.
> 
Hey, join the club. I've had a similar array, which were returning a
default inventory during bootup. Leaving us with no chance to detect
if the default inventory was the correct one or not.

I even posted a patch some time ago; if you wish I can drag it out.

> However, I'm still left wondering why we are adding PQ=1, PDT=0 devices in the sequential LUN scan at all.
> Are there media changer devices out there that we've seen respond like this? Even so, does it make sense
> to add PQ=1, PDT=0 LUNs for LUN > 0?
> 
Yes, unfortunately we need this. NetApp arrays have a habit of
returning 'PQ=1' for unconnected LUN 0, even though higher LUNs are
present. So we need to add devices for PQ=1, otherwise we wouldn't
be able to scan them.
We _might_ be able to tweak this by ignoring devices with PQ=1 and
LUN!=0; however, it might break other things.

As a net result, do avoid sequential scan if at all possible.
I would rather work on getting REPORT LUNs to reliably report data.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		               zSeries & Storage
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: SCSI scanning behavior

[Bart Van Assche]

On 08/26/15 06:02, Hannes Reinecke wrote:
> On 08/21/2015 12:11 AM, Brian King wrote:
>> However, I'm still left wondering why we are adding PQ=1, PDT=0 devices in the sequential LUN scan at all.
>> Are there media changer devices out there that we've seen respond like this? Even so, does it make sense
>> to add PQ=1, PDT=0 LUNs for LUN > 0?
>>
> Yes, unfortunately we need this. NetApp arrays have a habit of
> returning 'PQ=1' for unconnected LUN 0, even though higher LUNs are
> present. So we need to add devices for PQ=1, otherwise we wouldn't
> be able to scan them.
> We _might_ be able to tweak this by ignoring devices with PQ=1 and
> LUN!=0; however, it might break other things.

Hello Hannes,

The code in scsi_scan.c already skips LUNs with PQ=1 and PDT=0x1f. I'm 
not sure we should skip LUNS with PQ=1 and a PDT value other than 0x1f.

Thanks,

Bart.

[PATCH] SCSI: Scale up REPORT_LUNS timeout on failure

[Brian King]

This patch fixes an issue seen with an IBM 2145 (SVC) where, following an error
injection test which results in paths going offline, when they came
back online, the path would timeout the REPORT_LUNS issued during the
scan. This timeout situation continued until retries were expired, resulting in
falling back to a sequential LUN scan. Then, since the target responds
with PQ=1, PDT=0 for all possible LUNs, due to the way the sequential
LUN scan code works, we end up adding 512 LUNs for each target, when there
is really only a small handful of LUNs that are actually present.

This patch doubles the timeout used on the REPORT_LUNS for each retry
after a timeout is seen on a REPORT_LUNS. This patch solves the issue
of 512 non existent LUNs showing up after this event. Running the test
with this patch still showed that we were regularly hitting two timeouts,
but the third, and final, REPORT_LUNS was always successful.

Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
---

 drivers/scsi/scsi_scan.c |    5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff -puN drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalate drivers/scsi/scsi_scan.c
--- linux/drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalate	2015-09-02 08:49:07.268243497 -0500
+++ linux-bjking1/drivers/scsi/scsi_scan.c	2015-09-02 08:49:07.272243461 -0500
@@ -1304,6 +1304,7 @@ static int scsi_report_lun_scan(struct s
 	struct scsi_device *sdev;
 	struct Scsi_Host *shost = dev_to_shost(&starget->dev);
 	int ret = 0;
+	int timeout = SCSI_TIMEOUT + 4 * HZ;
 
 	/*
 	 * Only support SCSI-3 and up devices if BLIST_NOREPORTLUN is not set.
@@ -1383,7 +1384,7 @@ retry:
 
 		result = scsi_execute_req(sdev, scsi_cmd, DMA_FROM_DEVICE,
 					  lun_data, length, &sshdr,
-					  SCSI_TIMEOUT + 4 * HZ, 3, NULL);
+					  timeout, 3, NULL);
 
 		SCSI_LOG_SCAN_BUS(3, sdev_printk (KERN_INFO, sdev,
 				"scsi scan: REPORT LUNS"
@@ -1392,6 +1393,8 @@ retry:
 				retries, result));
 		if (result == 0)
 			break;
+		else if (host_byte(result) == DID_TIME_OUT)
+			timeout = timeout * 2;
 		else if (scsi_sense_valid(&sshdr)) {
 			if (sshdr.sense_key != UNIT_ATTENTION)
 				break;
_

Re: [PATCH] SCSI: Scale up REPORT_LUNS timeout on failure

[Bart Van Assche]

On 09/02/15 07:31, Brian King wrote:
>
> This patch fixes an issue seen with an IBM 2145 (SVC) where, following an error
> injection test which results in paths going offline, when they came
> back online, the path would timeout the REPORT_LUNS issued during the
> scan. This timeout situation continued until retries were expired, resulting in
> falling back to a sequential LUN scan. Then, since the target responds
> with PQ=1, PDT=0 for all possible LUNs, due to the way the sequential
> LUN scan code works, we end up adding 512 LUNs for each target, when there
> is really only a small handful of LUNs that are actually present.
>
> This patch doubles the timeout used on the REPORT_LUNS for each retry
> after a timeout is seen on a REPORT_LUNS. This patch solves the issue
> of 512 non existent LUNs showing up after this event. Running the test
> with this patch still showed that we were regularly hitting two timeouts,
> but the third, and final, REPORT_LUNS was always successful.
>
> Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
> ---
>
>   drivers/scsi/scsi_scan.c |    5 ++++-
>   1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff -puN drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalate drivers/scsi/scsi_scan.c
> --- linux/drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalate	2015-09-02 08:49:07.268243497 -0500
> +++ linux-bjking1/drivers/scsi/scsi_scan.c	2015-09-02 08:49:07.272243461 -0500
> @@ -1304,6 +1304,7 @@ static int scsi_report_lun_scan(struct s
>   	struct scsi_device *sdev;
>   	struct Scsi_Host *shost = dev_to_shost(&starget->dev);
>   	int ret = 0;
> +	int timeout = SCSI_TIMEOUT + 4 * HZ;
>
>   	/*
>   	 * Only support SCSI-3 and up devices if BLIST_NOREPORTLUN is not set.
> @@ -1383,7 +1384,7 @@ retry:
>
>   		result = scsi_execute_req(sdev, scsi_cmd, DMA_FROM_DEVICE,
>   					  lun_data, length, &sshdr,
> -					  SCSI_TIMEOUT + 4 * HZ, 3, NULL);
> +					  timeout, 3, NULL);
>
>   		SCSI_LOG_SCAN_BUS(3, sdev_printk (KERN_INFO, sdev,
>   				"scsi scan: REPORT LUNS"
> @@ -1392,6 +1393,8 @@ retry:
>   				retries, result));
>   		if (result == 0)
>   			break;
> +		else if (host_byte(result) == DID_TIME_OUT)
> +			timeout = timeout * 2;
>   		else if (scsi_sense_valid(&sshdr)) {
>   			if (sshdr.sense_key != UNIT_ATTENTION)
>   				break;

This is somewhat of a hack, but anyway:

Reviewed-by: Bart Van Assche <bvanassche@sandisk.com>

Re: [PATCH] SCSI: Scale up REPORT_LUNS timeout on failure

[Brian King]

On 09/04/2015 10:00 AM, Bart Van Assche wrote:
> On 09/02/15 07:31, Brian King wrote:
>>
>> This patch fixes an issue seen with an IBM 2145 (SVC) where, following an error
>> injection test which results in paths going offline, when they came
>> back online, the path would timeout the REPORT_LUNS issued during the
>> scan. This timeout situation continued until retries were expired, resulting in
>> falling back to a sequential LUN scan. Then, since the target responds
>> with PQ=1, PDT=0 for all possible LUNs, due to the way the sequential
>> LUN scan code works, we end up adding 512 LUNs for each target, when there
>> is really only a small handful of LUNs that are actually present.
>>
>> This patch doubles the timeout used on the REPORT_LUNS for each retry
>> after a timeout is seen on a REPORT_LUNS. This patch solves the issue
>> of 512 non existent LUNs showing up after this event. Running the test
>> with this patch still showed that we were regularly hitting two timeouts,
>> but the third, and final, REPORT_LUNS was always successful.
>>
>> Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
>> ---
>>
>>   drivers/scsi/scsi_scan.c |    5 ++++-
>>   1 file changed, 4 insertions(+), 1 deletion(-)
>>
>> diff -puN drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalate drivers/scsi/scsi_scan.c
>> --- linux/drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalate    2015-09-02 08:49:07.268243497 -0500
>> +++ linux-bjking1/drivers/scsi/scsi_scan.c    2015-09-02 08:49:07.272243461 -0500
>> @@ -1304,6 +1304,7 @@ static int scsi_report_lun_scan(struct s
>>       struct scsi_device *sdev;
>>       struct Scsi_Host *shost = dev_to_shost(&starget->dev);
>>       int ret = 0;
>> +    int timeout = SCSI_TIMEOUT + 4 * HZ;
>>
>>       /*
>>        * Only support SCSI-3 and up devices if BLIST_NOREPORTLUN is not set.
>> @@ -1383,7 +1384,7 @@ retry:
>>
>>           result = scsi_execute_req(sdev, scsi_cmd, DMA_FROM_DEVICE,
>>                         lun_data, length, &sshdr,
>> -                      SCSI_TIMEOUT + 4 * HZ, 3, NULL);
>> +                      timeout, 3, NULL);
>>
>>           SCSI_LOG_SCAN_BUS(3, sdev_printk (KERN_INFO, sdev,
>>                   "scsi scan: REPORT LUNS"
>> @@ -1392,6 +1393,8 @@ retry:
>>                   retries, result));
>>           if (result == 0)
>>               break;
>> +        else if (host_byte(result) == DID_TIME_OUT)
>> +            timeout = timeout * 2;
>>           else if (scsi_sense_valid(&sshdr)) {
>>               if (sshdr.sense_key != UNIT_ATTENTION)
>>                   break;
> 
> This is somewhat of a hack, but anyway:
> 
> Reviewed-by: Bart Van Assche <bvanassche@sandisk.com>

Agreed.

Thanks for the review.

-Brian

-- 
Brian King
Power Linux I/O
IBM Linux Technology Center

Re: [PATCH] SCSI: Scale up REPORT_LUNS timeout on failure

[James Bottomley]

On Wed, 2015-09-02 at 09:31 -0500, Brian King wrote:
> This patch fixes an issue seen with an IBM 2145 (SVC) where, following an error
> injection test which results in paths going offline, when they came
> back online, the path would timeout the REPORT_LUNS issued during the
> scan. This timeout situation continued until retries were expired, resulting in
> falling back to a sequential LUN scan. Then, since the target responds
> with PQ=1, PDT=0 for all possible LUNs, due to the way the sequential
> LUN scan code works, we end up adding 512 LUNs for each target, when there
> is really only a small handful of LUNs that are actually present.
> 
> This patch doubles the timeout used on the REPORT_LUNS for each retry
> after a timeout is seen on a REPORT_LUNS. This patch solves the issue
> of 512 non existent LUNs showing up after this event. Running the test
> with this patch still showed that we were regularly hitting two timeouts,
> but the third, and final, REPORT_LUNS was always successful.
> 
> Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
> ---
> 
>  drivers/scsi/scsi_scan.c |    5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff -puN drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalate drivers/scsi/scsi_scan.c
> --- linux/drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalate	2015-09-02 08:49:07.268243497 -0500
> +++ linux-bjking1/drivers/scsi/scsi_scan.c	2015-09-02 08:49:07.272243461 -0500
> @@ -1304,6 +1304,7 @@ static int scsi_report_lun_scan(struct s
>  	struct scsi_device *sdev;
>  	struct Scsi_Host *shost = dev_to_shost(&starget->dev);
>  	int ret = 0;
> +	int timeout = SCSI_TIMEOUT + 4 * HZ;
>  
>  	/*
>  	 * Only support SCSI-3 and up devices if BLIST_NOREPORTLUN is not set.
> @@ -1383,7 +1384,7 @@ retry:
>  
>  		result = scsi_execute_req(sdev, scsi_cmd, DMA_FROM_DEVICE,
>  					  lun_data, length, &sshdr,
> -					  SCSI_TIMEOUT + 4 * HZ, 3, NULL);
> +					  timeout, 3, NULL);
>  
>  		SCSI_LOG_SCAN_BUS(3, sdev_printk (KERN_INFO, sdev,
>  				"scsi scan: REPORT LUNS"
> @@ -1392,6 +1393,8 @@ retry:
>  				retries, result));
>  		if (result == 0)
>  			break;
> +		else if (host_byte(result) == DID_TIME_OUT)
> +			timeout = timeout * 2;
>  		else if (scsi_sense_valid(&sshdr)) {
>  			if (sshdr.sense_key != UNIT_ATTENTION)

Actually, this is a bit pointless, isn't it; why retry, why not just set
the initial timeout? ... I could understand if retrying and printing a
message gave important or useful information, but it doesn't.  How long
do you actually need? ... we can just up the initial timeout to that.
Currently we have a hacked 6s which looks arbitrary.  Would 15s be
better?  Nothing really times out anyway, so everything else will still
reply within the original 6s giving zero impact in the everyday case.

James

Re: [PATCH] SCSI: Scale up REPORT_LUNS timeout on failure

[Brian King]

On 09/04/2015 10:36 AM, James Bottomley wrote:
> On Wed, 2015-09-02 at 09:31 -0500, Brian King wrote:
>> This patch fixes an issue seen with an IBM 2145 (SVC) where, following an error
>> injection test which results in paths going offline, when they came
>> back online, the path would timeout the REPORT_LUNS issued during the
>> scan. This timeout situation continued until retries were expired, resulting in
>> falling back to a sequential LUN scan. Then, since the target responds
>> with PQ=1, PDT=0 for all possible LUNs, due to the way the sequential
>> LUN scan code works, we end up adding 512 LUNs for each target, when there
>> is really only a small handful of LUNs that are actually present.
>>
>> This patch doubles the timeout used on the REPORT_LUNS for each retry
>> after a timeout is seen on a REPORT_LUNS. This patch solves the issue
>> of 512 non existent LUNs showing up after this event. Running the test
>> with this patch still showed that we were regularly hitting two timeouts,
>> but the third, and final, REPORT_LUNS was always successful.
>>
>> Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
>> ---
>>
>>  drivers/scsi/scsi_scan.c |    5 ++++-
>>  1 file changed, 4 insertions(+), 1 deletion(-)
>>
>> diff -puN drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalate drivers/scsi/scsi_scan.c
>> --- linux/drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalate	2015-09-02 08:49:07.268243497 -0500
>> +++ linux-bjking1/drivers/scsi/scsi_scan.c	2015-09-02 08:49:07.272243461 -0500
>> @@ -1304,6 +1304,7 @@ static int scsi_report_lun_scan(struct s
>>  	struct scsi_device *sdev;
>>  	struct Scsi_Host *shost = dev_to_shost(&starget->dev);
>>  	int ret = 0;
>> +	int timeout = SCSI_TIMEOUT + 4 * HZ;
>>  
>>  	/*
>>  	 * Only support SCSI-3 and up devices if BLIST_NOREPORTLUN is not set.
>> @@ -1383,7 +1384,7 @@ retry:
>>  
>>  		result = scsi_execute_req(sdev, scsi_cmd, DMA_FROM_DEVICE,
>>  					  lun_data, length, &sshdr,
>> -					  SCSI_TIMEOUT + 4 * HZ, 3, NULL);
>> +					  timeout, 3, NULL);
>>  
>>  		SCSI_LOG_SCAN_BUS(3, sdev_printk (KERN_INFO, sdev,
>>  				"scsi scan: REPORT LUNS"
>> @@ -1392,6 +1393,8 @@ retry:
>>  				retries, result));
>>  		if (result == 0)
>>  			break;
>> +		else if (host_byte(result) == DID_TIME_OUT)
>> +			timeout = timeout * 2;
>>  		else if (scsi_sense_valid(&sshdr)) {
>>  			if (sshdr.sense_key != UNIT_ATTENTION)
> 
> Actually, this is a bit pointless, isn't it; why retry, why not just set
> the initial timeout? ... I could understand if retrying and printing a
> message gave important or useful information, but it doesn't.  How long
> do you actually need? ... we can just up the initial timeout to that.
> Currently we have a hacked 6s which looks arbitrary.  Would 15s be
> better?  Nothing really times out anyway, so everything else will still
> reply within the original 6s giving zero impact in the everyday case.

12 seconds definitely isn't long enough, but 24 seconds seems to work, at least
after we go through both a 6 and 12 second timeout. Anyone opposed to using 30 seconds?
15 seconds is likely to be right on the edge in this scenario.


-- 
Brian King
Power Linux I/O
IBM Linux Technology Center

Re: [PATCH] SCSI: Scale up REPORT_LUNS timeout on failure

[James Bottomley]

On Fri, 2015-09-04 at 10:47 -0500, Brian King wrote:
> On 09/04/2015 10:36 AM, James Bottomley wrote:
> > On Wed, 2015-09-02 at 09:31 -0500, Brian King wrote:
> >> This patch fixes an issue seen with an IBM 2145 (SVC) where, following an error
> >> injection test which results in paths going offline, when they came
> >> back online, the path would timeout the REPORT_LUNS issued during the
> >> scan. This timeout situation continued until retries were expired, resulting in
> >> falling back to a sequential LUN scan. Then, since the target responds
> >> with PQ=1, PDT=0 for all possible LUNs, due to the way the sequential
> >> LUN scan code works, we end up adding 512 LUNs for each target, when there
> >> is really only a small handful of LUNs that are actually present.
> >>
> >> This patch doubles the timeout used on the REPORT_LUNS for each retry
> >> after a timeout is seen on a REPORT_LUNS. This patch solves the issue
> >> of 512 non existent LUNs showing up after this event. Running the test
> >> with this patch still showed that we were regularly hitting two timeouts,
> >> but the third, and final, REPORT_LUNS was always successful.
> >>
> >> Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
> >> ---
> >>
> >>  drivers/scsi/scsi_scan.c |    5 ++++-
> >>  1 file changed, 4 insertions(+), 1 deletion(-)
> >>
> >> diff -puN drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalate drivers/scsi/scsi_scan.c
> >> --- linux/drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalate	2015-09-02 08:49:07.268243497 -0500
> >> +++ linux-bjking1/drivers/scsi/scsi_scan.c	2015-09-02 08:49:07.272243461 -0500
> >> @@ -1304,6 +1304,7 @@ static int scsi_report_lun_scan(struct s
> >>  	struct scsi_device *sdev;
> >>  	struct Scsi_Host *shost = dev_to_shost(&starget->dev);
> >>  	int ret = 0;
> >> +	int timeout = SCSI_TIMEOUT + 4 * HZ;
> >>  
> >>  	/*
> >>  	 * Only support SCSI-3 and up devices if BLIST_NOREPORTLUN is not set.
> >> @@ -1383,7 +1384,7 @@ retry:
> >>  
> >>  		result = scsi_execute_req(sdev, scsi_cmd, DMA_FROM_DEVICE,
> >>  					  lun_data, length, &sshdr,
> >> -					  SCSI_TIMEOUT + 4 * HZ, 3, NULL);
> >> +					  timeout, 3, NULL);
> >>  
> >>  		SCSI_LOG_SCAN_BUS(3, sdev_printk (KERN_INFO, sdev,
> >>  				"scsi scan: REPORT LUNS"
> >> @@ -1392,6 +1393,8 @@ retry:
> >>  				retries, result));
> >>  		if (result == 0)
> >>  			break;
> >> +		else if (host_byte(result) == DID_TIME_OUT)
> >> +			timeout = timeout * 2;
> >>  		else if (scsi_sense_valid(&sshdr)) {
> >>  			if (sshdr.sense_key != UNIT_ATTENTION)
> > 
> > Actually, this is a bit pointless, isn't it; why retry, why not just set
> > the initial timeout? ... I could understand if retrying and printing a
> > message gave important or useful information, but it doesn't.  How long
> > do you actually need? ... we can just up the initial timeout to that.
> > Currently we have a hacked 6s which looks arbitrary.  Would 15s be
> > better?  Nothing really times out anyway, so everything else will still
> > reply within the original 6s giving zero impact in the everyday case.
> 
> 12 seconds definitely isn't long enough, but 24 seconds seems to work, at least
> after we go through both a 6 and 12 second timeout. Anyone opposed to using 30 seconds?
> 15 seconds is likely to be right on the edge in this scenario.

30s is fine by me.  I think the initial 2s was from the sequential
inquiry scan so as not to wait too long.  The extra 4s was added because
that was too short for report luns on some devices; I suspect some
larger arrays take a while just to gather all the data.

30s is also the traditional rq_timeout, so it may be possible to re-use
this parameter.  Currently it's set up in the ULD, so it's zero unless
the slave_configure requested a special value.  Traditionally, it's the
timeout for _READ and _WRITE, not special commands, but it feels like
REPORT_LUNS should follow this timeout as well and it would give you a
configurable way of updating it in your driver.  If we do it this way,
you'd have to set it in slave_alloc, because slave_configure is too
late.

James

[PATCH] SCSI: Increase REPORT_LUNS timeout

[Brian King]

On 09/04/2015 11:15 AM, James Bottomley wrote:
> On Fri, 2015-09-04 at 10:47 -0500, Brian King wrote:
>> On 09/04/2015 10:36 AM, James Bottomley wrote:
>>> On Wed, 2015-09-02 at 09:31 -0500, Brian King wrote:
>>>> This patch fixes an issue seen with an IBM 2145 (SVC) where, following an error
>>>> injection test which results in paths going offline, when they came
>>>> back online, the path would timeout the REPORT_LUNS issued during the
>>>> scan. This timeout situation continued until retries were expired, resulting in
>>>> falling back to a sequential LUN scan. Then, since the target responds
>>>> with PQ=1, PDT=0 for all possible LUNs, due to the way the sequential
>>>> LUN scan code works, we end up adding 512 LUNs for each target, when there
>>>> is really only a small handful of LUNs that are actually present.
>>>>
>>>> This patch doubles the timeout used on the REPORT_LUNS for each retry
>>>> after a timeout is seen on a REPORT_LUNS. This patch solves the issue
>>>> of 512 non existent LUNs showing up after this event. Running the test
>>>> with this patch still showed that we were regularly hitting two timeouts,
>>>> but the third, and final, REPORT_LUNS was always successful.
>>>>
>>>> Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
>>>> ---
>>>>
>>>>  drivers/scsi/scsi_scan.c |    5 ++++-
>>>>  1 file changed, 4 insertions(+), 1 deletion(-)
>>>>
>>>> diff -puN drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalate drivers/scsi/scsi_scan.c
>>>> --- linux/drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalate	2015-09-02 08:49:07.268243497 -0500
>>>> +++ linux-bjking1/drivers/scsi/scsi_scan.c	2015-09-02 08:49:07.272243461 -0500
>>>> @@ -1304,6 +1304,7 @@ static int scsi_report_lun_scan(struct s
>>>>  	struct scsi_device *sdev;
>>>>  	struct Scsi_Host *shost = dev_to_shost(&starget->dev);
>>>>  	int ret = 0;
>>>> +	int timeout = SCSI_TIMEOUT + 4 * HZ;
>>>>  
>>>>  	/*
>>>>  	 * Only support SCSI-3 and up devices if BLIST_NOREPORTLUN is not set.
>>>> @@ -1383,7 +1384,7 @@ retry:
>>>>  
>>>>  		result = scsi_execute_req(sdev, scsi_cmd, DMA_FROM_DEVICE,
>>>>  					  lun_data, length, &sshdr,
>>>> -					  SCSI_TIMEOUT + 4 * HZ, 3, NULL);
>>>> +					  timeout, 3, NULL);
>>>>  
>>>>  		SCSI_LOG_SCAN_BUS(3, sdev_printk (KERN_INFO, sdev,
>>>>  				"scsi scan: REPORT LUNS"
>>>> @@ -1392,6 +1393,8 @@ retry:
>>>>  				retries, result));
>>>>  		if (result == 0)
>>>>  			break;
>>>> +		else if (host_byte(result) == DID_TIME_OUT)
>>>> +			timeout = timeout * 2;
>>>>  		else if (scsi_sense_valid(&sshdr)) {
>>>>  			if (sshdr.sense_key != UNIT_ATTENTION)
>>>
>>> Actually, this is a bit pointless, isn't it; why retry, why not just set
>>> the initial timeout? ... I could understand if retrying and printing a
>>> message gave important or useful information, but it doesn't.  How long
>>> do you actually need? ... we can just up the initial timeout to that.
>>> Currently we have a hacked 6s which looks arbitrary.  Would 15s be
>>> better?  Nothing really times out anyway, so everything else will still
>>> reply within the original 6s giving zero impact in the everyday case.
>>
>> 12 seconds definitely isn't long enough, but 24 seconds seems to work, at least
>> after we go through both a 6 and 12 second timeout. Anyone opposed to using 30 seconds?
>> 15 seconds is likely to be right on the edge in this scenario.
> 
> 30s is fine by me.  I think the initial 2s was from the sequential
> inquiry scan so as not to wait too long.  The extra 4s was added because
> that was too short for report luns on some devices; I suspect some
> larger arrays take a while just to gather all the data.
> 
> 30s is also the traditional rq_timeout, so it may be possible to re-use
> this parameter.  Currently it's set up in the ULD, so it's zero unless
> the slave_configure requested a special value.  Traditionally, it's the
> timeout for _READ and _WRITE, not special commands, but it feels like
> REPORT_LUNS should follow this timeout as well and it would give you a
> configurable way of updating it in your driver.  If we do it this way,
> you'd have to set it in slave_alloc, because slave_configure is too
> late.

I think we may just need to hard code it like the patch below. Here is the current flow for
setting this today:

slave_alloc
scsi scan: inquiry / report LUNs
slave_configure
sd attach

Some LLDDs set a default timeout in slave_configure today, so sd.c only sets a default timeout
if its not already set. It uses 30 seconds for disks and 75 seconds for optical devices.
If we start setting rq_timeout earlier, then the ULD will never know when it can set it.

Additionally, in this particular scenario, its not so much a case of behavior tied to the LLDD, its more tied
to the SCSI target. If there is concern about increasing the default to 30 seconds, we could
use a blist attribute for this.

-Brian

8<

This patch fixes an issue seen with an IBM 2145 (SVC) where, following an error
injection test which results in paths going offline, when they came
back online, the path would timeout the REPORT_LUNS issued during the
scan. This timeout situation continued until retries were expired, resulting in
falling back to a sequential LUN scan. Then, since the target responds
with PQ=1, PDT=0 for all possible LUNs, due to the way the sequential
LUN scan code works, we end up adding 512 LUNs for each target, when there
is really only a small handful of LUNs that are actually present.

This patch increases the timeout used on the REPORT_LUNS to 30 seconds.
This patch solves the issue of 512 non existent LUNs showing up after
this event.

Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
---

 drivers/scsi/scsi_scan.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff -puN drivers/scsi/scsi_scan.c~scsi_report_luns_30secs drivers/scsi/scsi_scan.c
--- linux/drivers/scsi/scsi_scan.c~scsi_report_luns_30secs	2015-09-04 14:38:47.890757391 -0500
+++ linux-bjking1/drivers/scsi/scsi_scan.c	2015-09-04 14:39:28.891459147 -0500
@@ -55,6 +55,7 @@
  * Default timeout
  */
 #define SCSI_TIMEOUT (2*HZ)
+#define SCSI_REPORT_LUNS_TIMEOUT (30*HZ)
 
 /*
  * Prefix values for the SCSI id's (stored in sysfs name field)
@@ -1383,7 +1384,7 @@ retry:
 
 		result = scsi_execute_req(sdev, scsi_cmd, DMA_FROM_DEVICE,
 					  lun_data, length, &sshdr,
-					  SCSI_TIMEOUT + 4 * HZ, 3, NULL);
+					  SCSI_REPORT_LUNS_TIMEOUT, 3, NULL);
 
 		SCSI_LOG_SCAN_BUS(3, sdev_printk (KERN_INFO, sdev,
 				"scsi scan: REPORT LUNS"
_

Re: [PATCH] SCSI: Increase REPORT_LUNS timeout

[Hannes Reinecke]

On 09/04/2015 09:47 PM, Brian King wrote:
> On 09/04/2015 11:15 AM, James Bottomley wrote:
>> On Fri, 2015-09-04 at 10:47 -0500, Brian King wrote:
>>> On 09/04/2015 10:36 AM, James Bottomley wrote:
>>>> On Wed, 2015-09-02 at 09:31 -0500, Brian King wrote:
>>>>> This patch fixes an issue seen with an IBM 2145 (SVC) where, following an error
>>>>> injection test which results in paths going offline, when they came
>>>>> back online, the path would timeout the REPORT_LUNS issued during the
>>>>> scan. This timeout situation continued until retries were expired, resulting in
>>>>> falling back to a sequential LUN scan. Then, since the target responds
>>>>> with PQ=1, PDT=0 for all possible LUNs, due to the way the sequential
>>>>> LUN scan code works, we end up adding 512 LUNs for each target, when there
>>>>> is really only a small handful of LUNs that are actually present.
>>>>>
>>>>> This patch doubles the timeout used on the REPORT_LUNS for each retry
>>>>> after a timeout is seen on a REPORT_LUNS. This patch solves the issue
>>>>> of 512 non existent LUNs showing up after this event. Running the test
>>>>> with this patch still showed that we were regularly hitting two timeouts,
>>>>> but the third, and final, REPORT_LUNS was always successful.
>>>>>
>>>>> Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
>>>>> ---
>>>>>
>>>>>  drivers/scsi/scsi_scan.c |    5 ++++-
>>>>>  1 file changed, 4 insertions(+), 1 deletion(-)
>>>>>
>>>>> diff -puN drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalate drivers/scsi/scsi_scan.c
>>>>> --- linux/drivers/scsi/scsi_scan.c~scsi_report_luns_timeout_escalate	2015-09-02 08:49:07.268243497 -0500
>>>>> +++ linux-bjking1/drivers/scsi/scsi_scan.c	2015-09-02 08:49:07.272243461 -0500
>>>>> @@ -1304,6 +1304,7 @@ static int scsi_report_lun_scan(struct s
>>>>>  	struct scsi_device *sdev;
>>>>>  	struct Scsi_Host *shost = dev_to_shost(&starget->dev);
>>>>>  	int ret = 0;
>>>>> +	int timeout = SCSI_TIMEOUT + 4 * HZ;
>>>>>  
>>>>>  	/*
>>>>>  	 * Only support SCSI-3 and up devices if BLIST_NOREPORTLUN is not set.
>>>>> @@ -1383,7 +1384,7 @@ retry:
>>>>>  
>>>>>  		result = scsi_execute_req(sdev, scsi_cmd, DMA_FROM_DEVICE,
>>>>>  					  lun_data, length, &sshdr,
>>>>> -					  SCSI_TIMEOUT + 4 * HZ, 3, NULL);
>>>>> +					  timeout, 3, NULL);
>>>>>  
>>>>>  		SCSI_LOG_SCAN_BUS(3, sdev_printk (KERN_INFO, sdev,
>>>>>  				"scsi scan: REPORT LUNS"
>>>>> @@ -1392,6 +1393,8 @@ retry:
>>>>>  				retries, result));
>>>>>  		if (result == 0)
>>>>>  			break;
>>>>> +		else if (host_byte(result) == DID_TIME_OUT)
>>>>> +			timeout = timeout * 2;
>>>>>  		else if (scsi_sense_valid(&sshdr)) {
>>>>>  			if (sshdr.sense_key != UNIT_ATTENTION)
>>>>
>>>> Actually, this is a bit pointless, isn't it; why retry, why not just set
>>>> the initial timeout? ... I could understand if retrying and printing a
>>>> message gave important or useful information, but it doesn't.  How long
>>>> do you actually need? ... we can just up the initial timeout to that.
>>>> Currently we have a hacked 6s which looks arbitrary.  Would 15s be
>>>> better?  Nothing really times out anyway, so everything else will still
>>>> reply within the original 6s giving zero impact in the everyday case.
>>>
>>> 12 seconds definitely isn't long enough, but 24 seconds seems to work, at least
>>> after we go through both a 6 and 12 second timeout. Anyone opposed to using 30 seconds?
>>> 15 seconds is likely to be right on the edge in this scenario.
>>
>> 30s is fine by me.  I think the initial 2s was from the sequential
>> inquiry scan so as not to wait too long.  The extra 4s was added because
>> that was too short for report luns on some devices; I suspect some
>> larger arrays take a while just to gather all the data.
>>
>> 30s is also the traditional rq_timeout, so it may be possible to re-use
>> this parameter.  Currently it's set up in the ULD, so it's zero unless
>> the slave_configure requested a special value.  Traditionally, it's the
>> timeout for _READ and _WRITE, not special commands, but it feels like
>> REPORT_LUNS should follow this timeout as well and it would give you a
>> configurable way of updating it in your driver.  If we do it this way,
>> you'd have to set it in slave_alloc, because slave_configure is too
>> late.
> 
> I think we may just need to hard code it like the patch below. Here is the current flow for
> setting this today:
> 
> slave_alloc
> scsi scan: inquiry / report LUNs
> slave_configure
> sd attach
> 
> Some LLDDs set a default timeout in slave_configure today, so sd.c only sets a default timeout
> if its not already set. It uses 30 seconds for disks and 75 seconds for optical devices.
> If we start setting rq_timeout earlier, then the ULD will never know when it can set it.
> 
> Additionally, in this particular scenario, its not so much a case of behavior tied to the LLDD, its more tied
> to the SCSI target. If there is concern about increasing the default to 30 seconds, we could
> use a blist attribute for this.
> 
> -Brian
> 
> 8<
> 
> This patch fixes an issue seen with an IBM 2145 (SVC) where, following an error
> injection test which results in paths going offline, when they came
> back online, the path would timeout the REPORT_LUNS issued during the
> scan. This timeout situation continued until retries were expired, resulting in
> falling back to a sequential LUN scan. Then, since the target responds
> with PQ=1, PDT=0 for all possible LUNs, due to the way the sequential
> LUN scan code works, we end up adding 512 LUNs for each target, when there
> is really only a small handful of LUNs that are actually present.
> 
> This patch increases the timeout used on the REPORT_LUNS to 30 seconds.
> This patch solves the issue of 512 non existent LUNs showing up after
> this event.
> 
> Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
> ---
> 
>  drivers/scsi/scsi_scan.c |    3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff -puN drivers/scsi/scsi_scan.c~scsi_report_luns_30secs drivers/scsi/scsi_scan.c
> --- linux/drivers/scsi/scsi_scan.c~scsi_report_luns_30secs	2015-09-04 14:38:47.890757391 -0500
> +++ linux-bjking1/drivers/scsi/scsi_scan.c	2015-09-04 14:39:28.891459147 -0500
> @@ -55,6 +55,7 @@
>   * Default timeout
>   */
>  #define SCSI_TIMEOUT (2*HZ)
> +#define SCSI_REPORT_LUNS_TIMEOUT (30*HZ)
>  
>  /*
>   * Prefix values for the SCSI id's (stored in sysfs name field)
> @@ -1383,7 +1384,7 @@ retry:
>  
>  		result = scsi_execute_req(sdev, scsi_cmd, DMA_FROM_DEVICE,
>  					  lun_data, length, &sshdr,
> -					  SCSI_TIMEOUT + 4 * HZ, 3, NULL);
> +					  SCSI_REPORT_LUNS_TIMEOUT, 3, NULL);
>  
>  		SCSI_LOG_SCAN_BUS(3, sdev_printk (KERN_INFO, sdev,
>  				"scsi scan: REPORT LUNS"
> _
> 
> 
That's far better.

Reviewed-by: Hannes Reinecke <hare@suse.de>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] SCSI: Increase REPORT_LUNS timeout

[Martin K. Petersen]

>>>>> "Brian" == Brian King <brking@linux.vnet.ibm.com> writes:

Brian> This patch increases the timeout used on the REPORT_LUNS to 30
Brian> seconds.  This patch solves the issue of 512 non existent LUNs
Brian> showing up after this event.

Applied.

-- 
Martin K. Petersen	Oracle Linux Engineering