[RFC][PATCH]SCSI signal(I/O) failure causes no response

[RFC][PATCH]SCSI signal(I/O) failure causes no response

[Masao Fukuchi]

Hi all,

We are planning to use linux for enterprise server.
Since the response time on hardware failure is important factor,
we are measuring the response time in case of the error by making
pseudo errors.

We generated some SCSI signal failure in the test.
When I cut the I/O signal during read command from host, 
I expected the command finished with error soon.
But the result was the command didn't finish forever(and no error
message was displayed).

I added some messages in order to investigate what has occurred.
>From the messages, the sequence was:

1.Fusion MPT driver issues read command to its firmware.
  (our server has LSI53C1030 as SCSI adapter)
  Then the firmware returns protocol error for the command.
  Fusion MPT driver makes DID_RESET status by protocol error 
  and sends it to SCSI midlayer.

2.SCSI midlayer analyzes the status from LLD.
  SCSI midlayer schedules command retry because the status is just
  DID_RESET status.
  (When the status has DID_RESET plus some sense code, the retry
   sequence depends on the sense code. But when the status has only
   DID_RESET, SCSI midlayer schedules command retry)

Sequence 1. and 2. are repeated infinitely and it causes no response.

To prevent this problem, I proposed Eric Moore to change the DID_RESET
status to DID_SOFT_ERROR in fusion MPT driver.
But he suggested me to change SCSI midlayer to prevent infinite loop.

So, I made following patch to prevent the problem.
(patch is based on kernel 2.6.5)

Comments welcome.

Thanks,
Masao Fukuchi

///
diff -uarN linux-2.6.5/drivers/scsi/scsi_lib.c linux-didreset/drivers/scsi/scsi_lib.c
--- linux-2.6.5/drivers/scsi/scsi_lib.c	2004-04-13 18:53:52.000000000 +0900
+++ linux-didreset/drivers/scsi/scsi_lib.c	2004-04-13 20:32:23.000000000 +0900
@@ -852,8 +852,10 @@
 		 * recovery reasons.  Just retry the request
 		 * and see what happens.  
 		 */
-		scsi_requeue_command(q, cmd);
-		return;
+		if (++cmd->retries < cmd->allowed) {
+			scsi_requeue_command(q, cmd);
+			return;
+		}
 	}
 	if (result) {
 		printk("SCSI error : <%d %d %d %d> return code = 0x%x\n",

RE: [RFC][PATCH]SCSI signal(I/O) failure causes no response

[Moore, Eric Dean]

The original request from Masao Fukuci was this:


 * than an unsolicited DID_ABORT.
 */
   sc->result = DID_RESET << 16;
+  if (++did_reset_cnt >= 5) {
+	sc->result = DID_SOFT_ERROR << 16;
+	did_reset_cnt = 0;
+  }


I wasn't too comfortable for the LLD to handle
retry count with DID_RESET, to transition to
DID_SOFT_ERROR. My thoughts were the DID_RESET logic
in scsi-mid layer should be handling retry count
to address infinite loop, as this same issue would
occur to any LLD.

Eric



On Tuesday, April 13, 2004 7:08 PM, Masao Fukuchi wrote:

> 
> 
> Hi all,
> 
> We are planning to use linux for enterprise server.
> Since the response time on hardware failure is important factor,
> we are measuring the response time in case of the error by making
> pseudo errors.
> 
> We generated some SCSI signal failure in the test.
> When I cut the I/O signal during read command from host, 
> I expected the command finished with error soon.
> But the result was the command didn't finish forever(and no error
> message was displayed).
> 
> I added some messages in order to investigate what has occurred.
> From the messages, the sequence was:
> 
> 1.Fusion MPT driver issues read command to its firmware.
>   (our server has LSI53C1030 as SCSI adapter)
>   Then the firmware returns protocol error for the command.
>   Fusion MPT driver makes DID_RESET status by protocol error 
>   and sends it to SCSI midlayer.
> 
> 2.SCSI midlayer analyzes the status from LLD.
>   SCSI midlayer schedules command retry because the status is just
>   DID_RESET status.
>   (When the status has DID_RESET plus some sense code, the retry
>    sequence depends on the sense code. But when the status has only
>    DID_RESET, SCSI midlayer schedules command retry)
> 
> Sequence 1. and 2. are repeated infinitely and it causes no response.
> 
> To prevent this problem, I proposed Eric Moore to change the DID_RESET
> status to DID_SOFT_ERROR in fusion MPT driver.
> But he suggested me to change SCSI midlayer to prevent infinite loop.
> 
> So, I made following patch to prevent the problem.
> (patch is based on kernel 2.6.5)
> 
> Comments welcome.
> 
> Thanks,
> Masao Fukuchi
> 
> ///
> diff -uarN linux-2.6.5/drivers/scsi/scsi_lib.c 
> linux-didreset/drivers/scsi/scsi_lib.c
> --- linux-2.6.5/drivers/scsi/scsi_lib.c	2004-04-13 
> 18:53:52.000000000 +0900
> +++ linux-didreset/drivers/scsi/scsi_lib.c	2004-04-13 
> 20:32:23.000000000 +0900
> @@ -852,8 +852,10 @@
>  		 * recovery reasons.  Just retry the request
>  		 * and see what happens.  
>  		 */
> -		scsi_requeue_command(q, cmd);
> -		return;
> +		if (++cmd->retries < cmd->allowed) {
> +			scsi_requeue_command(q, cmd);
> +			return;
> +		}
>  	}
>  	if (result) {
>  		printk("SCSI error : <%d %d %d %d> return code 
> = 0x%x\n",
> 

Re: [RFC][PATCH]SCSI signal(I/O) failure causes no response

[James Bottomley]

On Tue, 2004-04-13 at 20:07, Masao Fukuchi wrote:
> 1.Fusion MPT driver issues read command to its firmware.
>   (our server has LSI53C1030 as SCSI adapter)
>   Then the firmware returns protocol error for the command.
>   Fusion MPT driver makes DID_RESET status by protocol error 
>   and sends it to SCSI midlayer.
> 
> 2.SCSI midlayer analyzes the status from LLD.
>   SCSI midlayer schedules command retry because the status is just
>   DID_RESET status.
>   (When the status has DID_RESET plus some sense code, the retry
>    sequence depends on the sense code. But when the status has only
>    DID_RESET, SCSI midlayer schedules command retry)
> 
> Sequence 1. and 2. are repeated infinitely and it causes no response.
> 
> To prevent this problem, I proposed Eric Moore to change the DID_RESET
> status to DID_SOFT_ERROR in fusion MPT driver.
> But he suggested me to change SCSI midlayer to prevent infinite loop.

Well, there clearly is a problem, because we can't retry no-rety
commands that return DID_RESET (like tape commands or fastfail).

However, DID_RESET is supposed to be returned for events where it was
determined that the command was lost because the bus or device was reset
(either as part of error handling or because an external entity issued
the reset).  Since these events, if they originate externally, can be
beyond the control of the device, making retryable commands subject to
the retry limit would be asking for unnecessary I/O errors because of
something we couldn't do anything about.

Why is the LSI driver returning DID_RESET for the problem (i.e. is it
some type of external bus reset, or is something else going on)?

James

RE: [RFC][PATCH]SCSI signal(I/O) failure causes no response

[Moore, Eric Dean]

For this particular issue, Mr Masao Fukuchi has a
scsi bus test analyzer, in which he set the C/D signal 
to low during read operation.  The MPT firmware
returned MPI_IOCSTATUS_SCSI_PROTOCAL_ERROR, which from
the mpt manual means " An unrecoverable bus protocal error
as terminated the SCSI I/O" and the driver will 
return DID_RESET.

Here are some of the other cases which return
DID_RESET, however I doubt were returned with 
Mr Masao Fukuchi's test anaylzer:

MPI_IOCSTATUS_SCSI_TASK_TERMINATED
MPI_IOCSTATUS_SCSI_IOC_TERMINATED
MPI_IOCSTATUS_SCSI_EXT_TERMINATED



On Wednesday, April 14, 2004 9:28 AM, James Bottomley wrote:

> 
> 
> On Tue, 2004-04-13 at 20:07, Masao Fukuchi wrote:
> > 1.Fusion MPT driver issues read command to its firmware.
> >   (our server has LSI53C1030 as SCSI adapter)
> >   Then the firmware returns protocol error for the command.
> >   Fusion MPT driver makes DID_RESET status by protocol error 
> >   and sends it to SCSI midlayer.
> > 
> > 2.SCSI midlayer analyzes the status from LLD.
> >   SCSI midlayer schedules command retry because the status is just
> >   DID_RESET status.
> >   (When the status has DID_RESET plus some sense code, the retry
> >    sequence depends on the sense code. But when the status has only
> >    DID_RESET, SCSI midlayer schedules command retry)
> > 
> > Sequence 1. and 2. are repeated infinitely and it causes no 
> response.
> > 
> > To prevent this problem, I proposed Eric Moore to change 
> the DID_RESET
> > status to DID_SOFT_ERROR in fusion MPT driver.
> > But he suggested me to change SCSI midlayer to prevent 
> infinite loop.
> 
> Well, there clearly is a problem, because we can't retry no-rety
> commands that return DID_RESET (like tape commands or fastfail).
> 
> However, DID_RESET is supposed to be returned for events where it was
> determined that the command was lost because the bus or 
> device was reset
> (either as part of error handling or because an external entity issued
> the reset).  Since these events, if they originate externally, can be
> beyond the control of the device, making retryable commands subject to
> the retry limit would be asking for unnecessary I/O errors because of
> something we couldn't do anything about.
> 
> Why is the LSI driver returning DID_RESET for the problem (i.e. is it
> some type of external bus reset, or is something else going on)?
> 
> James
> 
> 

Re: [RFC][PATCH]SCSI signal(I/O) failure causes no response

[Masao Fukuchi]

I saw following IOC status by my test tool.
 MPI_IOCSTATUS_SCSI_TASK_TERMINATED
 MPI_IOCSTATUS_SCSI_IOC_TERMINATED

But for these IOC status, I saw some kind of error like another
IOC status or timeout just before(or after) the above IOC
status.

I think the SCSI_PROTOROL_ERROR has a some different meaning from
others.

Masao Fukuchi

Moore, Eric Dean wrote:
>For this particular issue, Mr Masao Fukuchi has a
>scsi bus test analyzer, in which he set the C/D signal 
>to low during read operation.  The MPT firmware
>returned MPI_IOCSTATUS_SCSI_PROTOCAL_ERROR, which from
>the mpt manual means " An unrecoverable bus protocal error
>as terminated the SCSI I/O" and the driver will 
>return DID_RESET.
>
>Here are some of the other cases which return
>DID_RESET, however I doubt were returned with 
>Mr Masao Fukuchi's test anaylzer:
>
>MPI_IOCSTATUS_SCSI_TASK_TERMINATED
>MPI_IOCSTATUS_SCSI_IOC_TERMINATED
>MPI_IOCSTATUS_SCSI_EXT_TERMINATED
>
>
>
>On Wednesday, April 14, 2004 9:28 AM, James Bottomley wrote:
>
>> 
>> 
>> On Tue, 2004-04-13 at 20:07, Masao Fukuchi wrote:
>> > 1.Fusion MPT driver issues read command to its firmware.
>> >   (our server has LSI53C1030 as SCSI adapter)
>> >   Then the firmware returns protocol error for the command.
>> >   Fusion MPT driver makes DID_RESET status by protocol error 
>> >   and sends it to SCSI midlayer.
>> > 
>> > 2.SCSI midlayer analyzes the status from LLD.
>> >   SCSI midlayer schedules command retry because the status is just
>> >   DID_RESET status.
>> >   (When the status has DID_RESET plus some sense code, the retry
>> >    sequence depends on the sense code. But when the status has only
>> >    DID_RESET, SCSI midlayer schedules command retry)
>> > 
>> > Sequence 1. and 2. are repeated infinitely and it causes no 
>> response.
>> > 
>> > To prevent this problem, I proposed Eric Moore to change 
>> the DID_RESET
>> > status to DID_SOFT_ERROR in fusion MPT driver.
>> > But he suggested me to change SCSI midlayer to prevent 
>> infinite loop.
>> 
>> Well, there clearly is a problem, because we can't retry no-rety
>> commands that return DID_RESET (like tape commands or fastfail).
>> 
>> However, DID_RESET is supposed to be returned for events where it was
>> determined that the command was lost because the bus or 
>> device was reset
>> (either as part of error handling or because an external entity issued
>> the reset).  Since these events, if they originate externally, can be
>> beyond the control of the device, making retryable commands subject to
>> the retry limit would be asking for unnecessary I/O errors because of
>> something we couldn't do anything about.
>> 
>> Why is the LSI driver returning DID_RESET for the problem (i.e. is it
>> some type of external bus reset, or is something else going on)?
>> 
>> James
>> 
>> 
>-
>To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [RFC][PATCH]SCSI signal(I/O) failure causes no response

[Moore, Eric Dean]

Which IOCSTATUS is returned when reproducing 
this issue that provides infinite loop?

Here are the meanings of IOCSTATUS from the manual:

MPI_IOCSTATUS_SCSI_TASK_TERMINATED - A SCSI Task Management request
has terminated I/O.

MPI_IOCSTATUS_SCSI_IOC_TERMINATED - A IOC has terminated the SCSI I/O.
This is typically an abort or bus reset initiated by the IOC.

MPI_IOCSTATUS_SCSI_EXT_TERMINATED - An external source has terminated the 
SCSI I/O. This is typically a bus reset from another initiator.

MPI_IOCSTATUS_SCSI_PROTOCAL_ERROR - An unrecoverable bus protocal error
has terminated the SCSI I/O.



On Wednesday, April 14, 2004 11:24 PM, Masao Fukuchi wrote:

> 
> 
> I saw following IOC status by my test tool.
>  MPI_IOCSTATUS_SCSI_TASK_TERMINATED
>  MPI_IOCSTATUS_SCSI_IOC_TERMINATED
> 
> But for these IOC status, I saw some kind of error like another
> IOC status or timeout just before(or after) the above IOC
> status.
> 
> I think the SCSI_PROTOROL_ERROR has a some different meaning from
> others.
> 
> Masao Fukuchi
> 
> Moore, Eric Dean wrote:
> >For this particular issue, Mr Masao Fukuchi has a
> >scsi bus test analyzer, in which he set the C/D signal 
> >to low during read operation.  The MPT firmware
> >returned MPI_IOCSTATUS_SCSI_PROTOCAL_ERROR, which from
> >the mpt manual means " An unrecoverable bus protocal error
> >as terminated the SCSI I/O" and the driver will 
> >return DID_RESET.
> >
> >Here are some of the other cases which return
> >DID_RESET, however I doubt were returned with 
> >Mr Masao Fukuchi's test anaylzer:
> >
> >MPI_IOCSTATUS_SCSI_TASK_TERMINATED
> >MPI_IOCSTATUS_SCSI_IOC_TERMINATED
> >MPI_IOCSTATUS_SCSI_EXT_TERMINATED
> >
> >
> >
> >On Wednesday, April 14, 2004 9:28 AM, James Bottomley wrote:
> >
> >> 
> >> 
> >> On Tue, 2004-04-13 at 20:07, Masao Fukuchi wrote:
> >> > 1.Fusion MPT driver issues read command to its firmware.
> >> >   (our server has LSI53C1030 as SCSI adapter)
> >> >   Then the firmware returns protocol error for the command.
> >> >   Fusion MPT driver makes DID_RESET status by protocol error 
> >> >   and sends it to SCSI midlayer.
> >> > 
> >> > 2.SCSI midlayer analyzes the status from LLD.
> >> >   SCSI midlayer schedules command retry because the 
> status is just
> >> >   DID_RESET status.
> >> >   (When the status has DID_RESET plus some sense code, the retry
> >> >    sequence depends on the sense code. But when the 
> status has only
> >> >    DID_RESET, SCSI midlayer schedules command retry)
> >> > 
> >> > Sequence 1. and 2. are repeated infinitely and it causes no 
> >> response.
> >> > 
> >> > To prevent this problem, I proposed Eric Moore to change 
> >> the DID_RESET
> >> > status to DID_SOFT_ERROR in fusion MPT driver.
> >> > But he suggested me to change SCSI midlayer to prevent 
> >> infinite loop.
> >> 
> >> Well, there clearly is a problem, because we can't retry no-rety
> >> commands that return DID_RESET (like tape commands or fastfail).
> >> 
> >> However, DID_RESET is supposed to be returned for events 
> where it was
> >> determined that the command was lost because the bus or 
> >> device was reset
> >> (either as part of error handling or because an external 
> entity issued
> >> the reset).  Since these events, if they originate 
> externally, can be
> >> beyond the control of the device, making retryable 
> commands subject to
> >> the retry limit would be asking for unnecessary I/O errors 
> because of
> >> something we couldn't do anything about.
> >> 
> >> Why is the LSI driver returning DID_RESET for the problem 
> (i.e. is it
> >> some type of external bus reset, or is something else going on)?
> >> 
> >> James
> >> 
> >> 
> >-
> >To unsubscribe from this list: send the line "unsubscribe 
> linux-scsi" in
> >the body of a message to majordomo@vger.kernel.org
> >More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: [RFC][PATCH]SCSI signal(I/O) failure causes no response

[Masao Fukuchi]

I checked my test report.

Moore, Eric Dean wrote:
>Which IOCSTATUS is returned when reproducing 
>this issue that provides infinite loop?
>
>Here are the meanings of IOCSTATUS from the manual:
>
>MPI_IOCSTATUS_SCSI_TASK_TERMINATED - A SCSI Task Management request
>has terminated I/O.

- no inifinite loop

>MPI_IOCSTATUS_SCSI_IOC_TERMINATED - A IOC has terminated the SCSI I/O.
>This is typically an abort or bus reset initiated by the IOC.

- cut ATN signal
  -> repeat SCSI_IOC_TERMINATED with LogInfo=0x11010100(bug! MID not found)
- cut DB signal
  -> repeat command timeout and SCSI_IOC_TERMINATED

>MPI_IOCSTATUS_SCSI_EXT_TERMINATED - An external source has terminated the 
>SCSI I/O. This is typically a bus reset from another initiator.

- I've not seen this message before.

>MPI_IOCSTATUS_SCSI_PROTOCAL_ERROR - An unrecoverable bus protocal error
>has terminated the SCSI I/O.

- cut I/O signal
  -> repeat SCSI_PROTOCOL_ERROR
- cut MSG signal
  -> repeat SCSI_PROTOCOL_ERROR

>
>On Wednesday, April 14, 2004 11:24 PM, Masao Fukuchi wrote:
>
>> 
>> 
>> I saw following IOC status by my test tool.
>>  MPI_IOCSTATUS_SCSI_TASK_TERMINATED
>>  MPI_IOCSTATUS_SCSI_IOC_TERMINATED
>> 
>> But for these IOC status, I saw some kind of error like another
>> IOC status or timeout just before(or after) the above IOC
>> status.
>> 
>> I think the SCSI_PROTOROL_ERROR has a some different meaning from
>> others.
>> 
>> Masao Fukuchi
>> 
>> Moore, Eric Dean wrote:
>> >For this particular issue, Mr Masao Fukuchi has a
>> >scsi bus test analyzer, in which he set the C/D signal 
>> >to low during read operation.  The MPT firmware
>> >returned MPI_IOCSTATUS_SCSI_PROTOCAL_ERROR, which from
>> >the mpt manual means " An unrecoverable bus protocal error
>> >as terminated the SCSI I/O" and the driver will 
>> >return DID_RESET.
>> >
>> >Here are some of the other cases which return
>> >DID_RESET, however I doubt were returned with 
>> >Mr Masao Fukuchi's test anaylzer:
>> >
>> >MPI_IOCSTATUS_SCSI_TASK_TERMINATED
>> >MPI_IOCSTATUS_SCSI_IOC_TERMINATED
>> >MPI_IOCSTATUS_SCSI_EXT_TERMINATED
>> >
>> >
>> >
>> >On Wednesday, April 14, 2004 9:28 AM, James Bottomley wrote:
>> >
>> >> 
>> >> 
>> >> On Tue, 2004-04-13 at 20:07, Masao Fukuchi wrote:
>> >> > 1.Fusion MPT driver issues read command to its firmware.
>> >> >   (our server has LSI53C1030 as SCSI adapter)
>> >> >   Then the firmware returns protocol error for the command.
>> >> >   Fusion MPT driver makes DID_RESET status by protocol error 
>> >> >   and sends it to SCSI midlayer.
>> >> > 
>> >> > 2.SCSI midlayer analyzes the status from LLD.
>> >> >   SCSI midlayer schedules command retry because the 
>> status is just
>> >> >   DID_RESET status.
>> >> >   (When the status has DID_RESET plus some sense code, the retry
>> >> >    sequence depends on the sense code. But when the 
>> status has only
>> >> >    DID_RESET, SCSI midlayer schedules command retry)
>> >> > 
>> >> > Sequence 1. and 2. are repeated infinitely and it causes no 
>> >> response.
>> >> > 
>> >> > To prevent this problem, I proposed Eric Moore to change 
>> >> the DID_RESET
>> >> > status to DID_SOFT_ERROR in fusion MPT driver.
>> >> > But he suggested me to change SCSI midlayer to prevent 
>> >> infinite loop.
>> >> 
>> >> Well, there clearly is a problem, because we can't retry no-rety
>> >> commands that return DID_RESET (like tape commands or fastfail).
>> >> 
>> >> However, DID_RESET is supposed to be returned for events 
>> where it was
>> >> determined that the command was lost because the bus or 
>> >> device was reset
>> >> (either as part of error handling or because an external 
>> entity issued
>> >> the reset).  Since these events, if they originate 
>> externally, can be
>> >> beyond the control of the device, making retryable 
>> commands subject to
>> >> the retry limit would be asking for unnecessary I/O errors 
>> because of
>> >> something we couldn't do anything about.
>> >> 
>> >> Why is the LSI driver returning DID_RESET for the problem 
>> (i.e. is it
>> >> some type of external bus reset, or is something else going on)?
>> >> 
>> >> James
>> >> 
>> >> 
>> >-
>> >To unsubscribe from this list: send the line "unsubscribe 
>> linux-scsi" in
>> >the body of a message to majordomo@vger.kernel.org
>> >More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> 

Re: [RFC][PATCH]SCSI signal(I/O) failure causes no response

[Jeremy Higdon]

On Wed, Apr 14, 2004 at 11:56:20AM -0400, Moore, Eric Dean wrote:
> For this particular issue, Mr Masao Fukuchi has a
> scsi bus test analyzer, in which he set the C/D signal 
> to low during read operation.  The MPT firmware
> returned MPI_IOCSTATUS_SCSI_PROTOCAL_ERROR, which from
> the mpt manual means " An unrecoverable bus protocal error
> as terminated the SCSI I/O" and the driver will 
> return DID_RESET.
> 
> Here are some of the other cases which return
> DID_RESET, however I doubt were returned with 
> Mr Masao Fukuchi's test anaylzer:
> 
> MPI_IOCSTATUS_SCSI_TASK_TERMINATED
> MPI_IOCSTATUS_SCSI_IOC_TERMINATED
> MPI_IOCSTATUS_SCSI_EXT_TERMINATED


I think I would return DID_ERROR for things like that as a sort
of generic "something-went-wrong" on the SCSI bus.

jeremy