Re: error handler scheduling

[Ren Mingxin <renmx@cn.fujitsu.com>]

On 03/29/2013 12:02 AM, Elliott, Robert (Server Storage) wrote:
> There are several possible reasons for SCSI command timeouts:
>      a) the command request did not get to the SCSI target port and logical
>         unit (e.g., error on the wire)
>      b) logical unit is still working on the command
>      c) the command completed, but status didn't get to the SCSI initiator port
>         and application client (e.g., error on the wire)
>
> SCSI doesn't have a good way to detect case (c). For status delivery errors
> detected by the logical unit, I once proposed that the logical unit establish
> a unit attention condition and record the status delivery problem in a log
> page (T10 proposal 04-072) but this proposal didn't draw much interest. The
> QUERY TASK task management function can detect case (b) vs. the other cases.
>
> With SSDs, a lengthy timeout derived from ancient SCSI floppy drives doesn't
> make sense. Timeouts should scale automatically based on the device type
> (e.g., use microseconds for SSDs and seconds for HDDs). The REPORT
> SUPPORTED OPERATION CODES command provides some command timeout values
> to facilitate this.
>
> For Base feature set drives I'm encouraging an approach like this for
> handling command timeouts:
>
> 1) at discovery time:
>      1a) send REPORT SUPPORTED OPERATION CODES to determine the nominal
>          and maximum command timeouts
>      1b) send REPORT SUPPORTED TASK MANAGEMENT FUNCTION to determine
>          the TMF timeouts
>
> 2) send the command (e.g., READ, WRITE, FORMAT UNIT, ...)
>
> If status arrives for the command at any time, exit out of this procedure.
> If an I_T nexus loss occurs, then that handling overrides this procedure
> as well. Otherwise:
>
> 3) if the nominal command timeout is long (e.g., for a command like FORMAT
> UNIT with IMMED=0, but not for IO commands like READ and WRITE), then wait
> a short time and send QUERY TASK to ensure the command got there:
>      3a) if the command is not there (probably lost in delivery, but
>          possibly lost status), go to step (2) to resend the command
>      3b) if the command is still being processed, keep waiting
>
> 4) if the nominal command timeout is reached, send QUERY TASK to determine
> what is happening:
>      4a) if the command is not there (if step (3) was run, then this
>          probably means lost status), go to step (2) to resend the command
>      4b) if the command is still being processed, keep waiting
>
> 5) if the maximum command timeout is reached, send QUERY TASK to determine
> what is happening:
>      5a) if the command is not there (since step (4) was run, this
>           probably means lost status), go to step (2) to resend the command
>      5b) if the command is still being processed, proceed to step (6)
>          to abort the command
>
> 6) send ABORT TASK to abort the command
>
> 7) If ABORT TASK succeeds, either:
>      7a) escalate to a stronger TMF or hard reset if this command
>         keeps having repeated problems; or
>      7b) go to step (2) to resend the command
>
> 8) If the ABORT TASK timeout is reached, either:
>      8a) escalate to a stronger TMF or hard reset, then go to step (2)
>          to resend the command; or
>      8b) declare the logical unit is unavailable
>
> Doug: for ***, In addition to WSNZ bit now letting the drive not support
> the value of zero, T10 proposal 13-052 changes WRITE SAME so the NUMBER
> OF LOGICAL BLOCKS set to zero (if supported) must honor the MAXIMUM WRITE
> SAME LENGTH field, so the drive can provide a reasonable timeout value
> for the command (not worry that the entire capacity might be specified).

Please let me summarize what this thread has talked about the scsi
eh latency:

1) some scsi cmds' timemout values are inappropriate, we can avoid
    timeout by:
    a) sg_format sets the IMMED bit and use TEST UNIT READY or REQUEST
       SENSE polling to monitor - by Douglas
    b) cut big cmd into some reasonable-sized ones - by Douglas
    c) improve timeout values according to device types - by Elliott
2) call ->done() on the command after lun reset - by Hannes

And, my question is:
- could we wake up eh thread ASAP instead of waiting for all cmds
   complete to fast scheduling?

BTW: my original question is here:
http://www.spinics.net/lists/linux-scsi/msg65107.html

Thanks,
Ren

> ---
> Rob Elliott    HP Server Storage
>
>
>
>> -----Original Message-----
>> From: linux-scsi-owner@vger.kernel.org [mailto:linux-scsi-
>> owner@vger.kernel.org] On Behalf Of Douglas Gilbert
>> Sent: Wednesday, 27 March, 2013 9:39 AM
>> To: James.Smart@emulex.com
>> Cc: linux-scsi@vger.kernel.org
>> Subject: Re: error handler scheduling
>>
>> On 13-03-26 10:11 PM, James Smart wrote:
>>> In looking through the error handler, if a command times out and is added to
>> the
>>> eh_cmd_q for the shost, the error handler is only awakened once shost-
>>> host_busy
>>> (total number of i/os posted to the shost) is equal to shost->host_failed
>>> (number of i/o that have been failed and put on the eh_cmd_q).  Which
>> means, any
>>> other i/o that was outstanding must either complete or have their timeout
>> fire.
>>> Additionally, as all further i/o is held off at the block layer as the shost is
>>> in recovery, new i/o cannot be submitted until the error handler runs and
>>> resolves the errored i/os.
>>>
>>> Is this true ?
>>>
>>> I take it is also true that the midlayer thus expects every i/o to have an i/o
>>> timeout.  True ?
>>>
>>> The crux of this point is that when the recovery thread runs to aborts the
>> timed
>>> out i/os, is at the mercy of the last command to complete or timeout.
>>> Additionally, as all further i/o is held off at the block layer as the shost is
>>> in recovery, new i/o cannot be submitted until the error handler runs and
>>> resolves the errored i/os. So all I/O on the host is stopped until that last i/o
>>> completes/times out.   The timeouts may be eons later.  Consider SCSI format
>>> commands or verify commands that can take hours to complete.
>>>
>>> Specifically, I'm in a situation currently, where an application is using sg to
>>> send a command to a target. The app selected no-timeout - by setting
>> timeout to
>>> MAX_INT. Effectively it's so large its infinite. This I/O was one of those
>>> "lost" on the storage fabric. There was another command that long ago timed
>> out
>>> and is sitting on the error handlers queue. But nothing is happening - new i/o,
>>> or error handler to resolve the failed i/o, until that inifinite i/o completes.
>>>
>>> I'm hoping I hear that I just misunderstand things.  If not,  is there a
>>> suggestion for how to resolve this predicament ?    IMHO, I'm surprised we
>> stop
>>> all i/o for error handling, and that it can be so long later... I would assume
>>> there's a minimum bound we would wait in the error handler (30s?) before
>> we
>>> unconditionally run it and abort anything that was outstanding.
>> James,
>> After many encounters with the Linux SCSI mid-level error
>> handler I have concluded it is uncontrollable and
>> seemingly random, seen from the user space. Interestingly,
>> several attempts to add finer grained controls over
>> lu/target/host resets have been rebuffed.
>>
>> So my policy is to avoid timeout induced resets (like the
>> plague). Hence the default with sg_format is to set the IMMED
>> bit and use TEST UNIT READY or REQUEST SENSE polling to
>> monitor progress **. With commands like VERIFY, send many
>> reasonably sized commands, not one big one. And a special
>> mention for the SCSI WRITE SAME command which probably
>> has T10's silliest definition: if the NUMBER OF
>> LOGICAL BLOCKS field is set to zero it means keep writing
>> until the end of the disk *** and that might be 20 hours
>> later! The equivalent field set to zero in a SCSI VERIFY
>> or WRITE *** command means do nothing.
>>
>> Doug Gilbert
>>
>>
>> **   You can still run into problems when a SCSI FORMAT UNIT
>>        with the IMMED bit set: some other kernel subsystem or
>>        user space program may decide to send a SCSI command to the
>>        disk during format. Then said code may not comprehend why
>>        the disk in question is not ready and ends up triggering
>>        mid-level error handling which blows the format out of
>>        the water. That leaves the disk in the "format corrupt"
>>        state.
>>
>> ***  recently the Block Limits VPD has (knee-)capped this
>>        with the WSNZ bit
>>
>> **** apart from the obsolete WRITE(6) command which found
>>        another non obvious interpretation for a zero transfer
>>        length
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>