[PATCH] libata: reduce ATA command timeout to 7secs

[PATCH] libata: reduce ATA command timeout to 7secs

[Tejun Heo]

Both ATA and ATAPI devices used the default timeouts defined by SCSI
high level driver.  For both disks and ODDs, it was 30secs, which was
way too long for disks.  This patch makes most ATA commands time out
after 7secs - the de facto ATA command timeout, while leaving ATAPI
timeout at 30secs.

Note that both normal commands and EH commands timeouts are adjusted
to 7 secs, but cache flushses still have 30sec timeout.  This is a
side effect of the way sd makes use of sdev->timeout, but this is a
good side effect in that ATA spec requires cache flushes are given
longer timeout.

Signed-off-by: Tejun Heo <htejun@gmail.com>
---
I think we've waited enough.  Combined with EH updates which will soon
be posted, this should make life much easier (well, at least
responsive) when something goes wrong with ATA.

 drivers/ata/libata-core.c |    2 +-
 drivers/ata/libata-scsi.c |    6 ++++++
 include/linux/libata.h    |    4 ++--
 3 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/drivers/ata/libata-core.c b/drivers/ata/libata-core.c
index 582e44d..a441c75 100644
--- a/drivers/ata/libata-core.c
+++ b/drivers/ata/libata-core.c
@@ -89,7 +89,7 @@ int libata_fua = 0;
 module_param_named(fua, libata_fua, int, 0444);
 MODULE_PARM_DESC(fua, "FUA support (0=off, 1=on)");
 
-static int ata_probe_timeout = ATA_TMOUT_INTERNAL / HZ;
+static int ata_probe_timeout = ATA_TMOUT_CMD / HZ;
 module_param(ata_probe_timeout, int, 0444);
 MODULE_PARM_DESC(ata_probe_timeout, "Set ATA probing timeout (seconds)");
 
diff --git a/drivers/ata/libata-scsi.c b/drivers/ata/libata-scsi.c
index 0009818..dc2157c 100644
--- a/drivers/ata/libata-scsi.c
+++ b/drivers/ata/libata-scsi.c
@@ -903,6 +903,12 @@ static void ata_scsi_dev_config(struct scsi_device *sdev,
 		depth = min(ATA_MAX_QUEUE - 1, depth);
 		scsi_adjust_queue_depth(sdev, MSG_SIMPLE_TAG, depth);
 	}
+
+	/* setup command timeout */
+	if (dev->class == ATA_DEV_ATA)
+		sdev->timeout = ATA_TMOUT_CMD;
+	else
+		sdev->timeout = ATAPI_TMOUT_CMD;
 }
 
 /**
diff --git a/include/linux/libata.h b/include/linux/libata.h
index 305e95f..4c3ed59 100644
--- a/include/linux/libata.h
+++ b/include/linux/libata.h
@@ -214,8 +214,8 @@ enum {
 	/* various lengths of time */
 	ATA_TMOUT_BOOT		= 30 * HZ,	/* heuristic */
 	ATA_TMOUT_BOOT_QUICK	= 7 * HZ,	/* heuristic */
-	ATA_TMOUT_INTERNAL	= 30 * HZ,
-	ATA_TMOUT_INTERNAL_QUICK = 5 * HZ,
+	ATA_TMOUT_CMD		= 7 * HZ,	/* de facto ATA cmd timeout */
+	ATAPI_TMOUT_CMD		= 30 * HZ,
 
 	/* ATA bus states */
 	BUS_UNKNOWN		= 0,
-- 
1.4.4.4

Re: [PATCH] libata: reduce ATA command timeout to 7secs

[Alan]

> Note that both normal commands and EH commands timeouts are adjusted
> to 7 secs, but cache flushses still have 30sec timeout.  This is a
> side effect of the way sd makes use of sdev->timeout, but this is a
> good side effect in that ATA spec requires cache flushes are given
> longer timeout.

Our cache flush handling still appears to be questionable if not wrong.
Not sure how SCSI expects it to behave but for ATA we should call it
again on a fatal media error.


ACK to the patch itself.

Re: [PATCH] libata: reduce ATA command timeout to 7secs

[Tejun Heo]

Alan wrote:
>> Note that both normal commands and EH commands timeouts are adjusted
>> to 7 secs, but cache flushses still have 30sec timeout.  This is a
>> side effect of the way sd makes use of sdev->timeout, but this is a
>> good side effect in that ATA spec requires cache flushes are given
>> longer timeout.
> 
> Our cache flush handling still appears to be questionable if not wrong.
> Not sure how SCSI expects it to behave but for ATA we should call it
> again on a fatal media error.

Planning on doing that soon.  Thanks.

-- 
tejun

Re: [PATCH] libata: reduce ATA command timeout to 7secs

[Ric Wheeler]

Tejun Heo wrote:

>Both ATA and ATAPI devices used the default timeouts defined by SCSI
>high level driver.  For both disks and ODDs, it was 30secs, which was
>way too long for disks.  This patch makes most ATA commands time out
>after 7secs - the de facto ATA command timeout, while leaving ATAPI
>timeout at 30secs.
>
>Note that both normal commands and EH commands timeouts are adjusted
>to 7 secs, but cache flushses still have 30sec timeout.  This is a
>side effect of the way sd makes use of sdev->timeout, but this is a
>good side effect in that ATA spec requires cache flushes are given
>longer timeout.
>
>
>  
>
I am not sure that this is really a requirement that ATA (or S-ATA) 
drives can meet.  There are definitely error conditions that will exceed 
7 seconds and take that 30 second time to respond.

If the drive is stuck in recovery mode and we retry too soon, we are 
going to be likely to write off a perfectly good disk which would be a 
huge downside ;-)

ric

Re: [PATCH] libata: reduce ATA command timeout to 7secs

[Tejun Heo]

Ric Wheeler wrote:
> 
> Tejun Heo wrote:
> 
>> Both ATA and ATAPI devices used the default timeouts defined by SCSI
>> high level driver.  For both disks and ODDs, it was 30secs, which was
>> way too long for disks.  This patch makes most ATA commands time out
>> after 7secs - the de facto ATA command timeout, while leaving ATAPI
>> timeout at 30secs.
>>
>> Note that both normal commands and EH commands timeouts are adjusted
>> to 7 secs, but cache flushses still have 30sec timeout.  This is a
>> side effect of the way sd makes use of sdev->timeout, but this is a
>> good side effect in that ATA spec requires cache flushes are given
>> longer timeout.
>>
>>
>>  
>>
> I am not sure that this is really a requirement that ATA (or S-ATA)
> drives can meet.  There are definitely error conditions that will exceed
> 7 seconds and take that 30 second time to respond.
> 
> If the drive is stuck in recovery mode and we retry too soon, we are
> going to be likely to write off a perfectly good disk which would be a
> huge downside ;-)

Hmm... I'm hearing different stories here.  Some say 7sec is good enough
and the supporting argument that the other os is using 7 sec timeout is
pretty convincing.

After command timeout, EH will kick in, reset and revalidate then return
the control to SCSI HLD, sd in this case which will retry the command
several times.  Would the reset make the drive to exit recovery mode
such that it has to start all over again when the command is retried?
Or does the drive stop responding to resets while in recovery mode?  The
latter would be fine.  libata gives and will continue to give more than
30 secs for drive to respond to reset.

Ideas?

-- 
tejun

Re: [PATCH] libata: reduce ATA command timeout to 7secs

[Alan]

> Hmm... I'm hearing different stories here.  Some say 7sec is good enough
> and the supporting argument that the other os is using 7 sec timeout is
> pretty convincing.

Firstly what is your goal ?

> 
> After command timeout, EH will kick in, reset and revalidate then return
> the control to SCSI HLD, sd in this case which will retry the command
> several times.  Would the reset make the drive to exit recovery mode
> such that it has to start all over again when the command is retried?
> Or does the drive stop responding to resets while in recovery mode?  The
> latter would be fine.  libata gives and will continue to give more than
> 30 secs for drive to respond to reset.
> 
> Ideas?

Some thinking based on discussion so far:

Give it 7 seconds for a command to complete. At that point you need to
have a chat with the drive since its probably gone AWOL. If you get an
error within 7 seconds then its different.

For a reported error within 7 seconds (CRC excepted)
	 do fast error cleanup (not a reset) when we can
	 if need be reset

On the first media error/timeout , retry the command but with the segment
list inverted
On the second reset retry each block individually

User specified time (allowing for recovery/queueing time) later, if we
haven't got the drive back running then give up on those blocks.

The reason I suggest issuing the segment list inverted is that its
statistically likely we only hit one bad area. It's also likey that bad
area is adjacent blocks (it might not be for some failures I grant). So
if we retry from the other end of the list we have a good chance that a
typical list of blocks (say 1-4, 300-305, 380-383, 440-456, 511, 535)
that blows up in the middle somewhere is going to run through the other
good segments and get them on disk or off disk and stuff up and going
again before stalling at the other end of the faulty area.

Alan

Re: [PATCH] libata: reduce ATA command timeout to 7secs

[Mark Lord]

Alan wrote:
>..
> Give it 7 seconds for a command to complete. At that point you need to
> have a chat with the drive since its probably gone AWOL. If you get an
> error within 7 seconds then its different.

7 seconds is not enough for current drives to report back.
Adding another (8 seconds total) is enough, but I prefer to
see a little margin there, so call it 10 seconds.

Cheers

Re: [PATCH] libata: reduce ATA command timeout to 7secs

[Eric D. Mudama]

On 2/2/07, Mark Lord <liml@rtr.ca> wrote:
> 7 seconds is not enough for current drives to report back.
> Adding another (8 seconds total) is enough, but I prefer to
> see a little margin there, so call it 10 seconds.

If you're going to allow 30 seconds (or more) for a cache flush, you
probably want to allow 30 seconds (or more) for any command that might
implicitly cause a cache flush.

Things like EXECUTE DEVICE DIAGNOSTICS, IDENTIFY DEVICE, NOP,
STANDBY/IDLE IMMEDIATE, both "soft" and "hard" reset, most SMART
commands, etc.

In fact, my understanding is that many devices will flush their write
caches to disk when receiving non-data commands, so those commands may
take a while if they occur immediately following heavy write activity,
even without errors being present.

While sure, a reset should function fine being issued in the middle of
a command, and bring the drive back to a ready state, I don't know if
these assumptions about 7s vs 8s vs 10s are always going to be valid
for disk drives purchased through distribution channels or in the
generic PC market, as opposed to "RAID edition" or "MaxLine" or
similar branding from other vendors, where the firmware may have been
specifically optimized for quicker command completion, quicker status
reporting and less exhaustive error recovery.  Many linux users I
assume are buying/borrowing/stealing cheap old gear from whatever
source they can, and I don't want to unnecessarilly risk compounding
the issue in their environments.

--eric

Re: [PATCH] libata: reduce ATA command timeout to 7secs

[Alan]

> generic PC market, as opposed to "RAID edition" or "MaxLine" or
> similar branding from other vendors, where the firmware may have been
> specifically optimized for quicker command completion, quicker status
> reporting and less exhaustive error recovery.  Many linux users I

Is there a way to detect RAID drives with this kind of timing, or does
someone need to take a baseball bat into the standards committee meeting
and add an identify field that gives command and "long command" timeouts
in seconds ?

If we can detect the "fast fail" drives (initially by say ident string
components) lets do it and the people who care can buy the right product
for the job.

Alan

Re: [PATCH] libata: reduce ATA command timeout to 7secs

[Eric D. Mudama]

On 2/2/07, Alan <alan@lxorguk.ukuu.org.uk> wrote:
> Is there a way to detect RAID drives with this kind of timing, or does
> someone need to take a baseball bat into the standards committee meeting
> and add an identify field that gives command and "long command" timeouts
> in seconds ?
>
> If we can detect the "fast fail" drives (initially by say ident string
> components) lets do it and the people who care can buy the right product
> for the job.

Like many new features in disk drives or other products, they're often
driven by a single or small set of large, paying customers to prove
them out prior to being standardized.  Given time, most eventually
settle on common interfaces to reduce costs.

This is a continuous process, the "concept -> development -> one
customer -> standard feature" path.  Unfortunately, you're right, it's
very difficult for individuals to interact with some of the standards
committees, and the exact working details (including detection) of
these features are often not made public for competitive reasons in
their initial offerings.

--eric

Re: [PATCH] libata: reduce ATA command timeout to 7secs

[Ric Wheeler]

Alan wrote:
>> generic PC market, as opposed to "RAID edition" or "MaxLine" or
>> similar branding from other vendors, where the firmware may have been
>> specifically optimized for quicker command completion, quicker status
>> reporting and less exhaustive error recovery.  Many linux users I
> 
> Is there a way to detect RAID drives with this kind of timing, or does
> someone need to take a baseball bat into the standards committee meeting
> and add an identify field that gives command and "long command" timeouts
> in seconds ?

We figure it out by testing lots & lots of drives ;-)

> 
> If we can detect the "fast fail" drives (initially by say ident string
> components) lets do it and the people who care can buy the right product
> for the job.

I don't think that there is any kind of feature field...

> 
> Alan
> 

Re: [PATCH] libata: reduce ATA command timeout to 7secs

[Tejun Heo]

Alan wrote:
>> Hmm... I'm hearing different stories here.  Some say 7sec is good enough
>> and the supporting argument that the other os is using 7 sec timeout is
>> pretty convincing.
> 
> Firstly what is your goal ?

Reducing timeout.  Timeouts occur much more frequently with SATA than
PATA.  Reducing timeout generally helps making the system more usable
and it also helps a lot if PMP is attached.  Many drives share a port
and certain PHY events make PMP lock up (at least current generation of
them).  This affects all devices attached to the PMP and with all the
extra links PHY events are much more likely, so it's pretty important to
be able to recover fast.

>> After command timeout, EH will kick in, reset and revalidate then return
>> the control to SCSI HLD, sd in this case which will retry the command
>> several times.  Would the reset make the drive to exit recovery mode
>> such that it has to start all over again when the command is retried?
>> Or does the drive stop responding to resets while in recovery mode?  The
>> latter would be fine.  libata gives and will continue to give more than
>> 30 secs for drive to respond to reset.
>>
>> Ideas?
> 
> Some thinking based on discussion so far:
> 
> Give it 7 seconds for a command to complete. At that point you need to
> have a chat with the drive since its probably gone AWOL. If you get an
> error within 7 seconds then its different.
> 
> For a reported error within 7 seconds (CRC excepted)
> 	 do fast error cleanup (not a reset) when we can
> 	 if need be reset

We're already doing the above.  On device errors, we just revalidate the
device.

> On the first media error/timeout , retry the command but with the segment
> list inverted

That certainly sounds interesting.

> On the second reset retry each block individually
>
> User specified time (allowing for recovery/queueing time) later, if we
> haven't got the drive back running then give up on those blocks.
> 
> The reason I suggest issuing the segment list inverted is that its
> statistically likely we only hit one bad area. It's also likey that bad
> area is adjacent blocks (it might not be for some failures I grant). So
> if we retry from the other end of the list we have a good chance that a
> typical list of blocks (say 1-4, 300-305, 380-383, 440-456, 511, 535)
> that blows up in the middle somewhere is going to run through the other
> good segments and get them on disk or off disk and stuff up and going
> again before stalling at the other end of the faulty area.

Yeap, that makes a lot of sense.  I think we need to trade off here.
How much recovery time and code are we gonna spend on recovering media
errors?  After certain point, it doesn't make sense.  Stalling disk
access for minutes to single out a few bad blocks in 4.7G dvd iso
doesn't really do any good.

Another thing to consider is that, IIRC, blk layer handles errors
bio-by-bio basis.  It doesn't matter whether one block failed or all of
them failed, any failure in a bio makes the whole bio marked bad, so
investing a lot of effort at SCSI/libata layer to find out exactly which
blocks fail is rather fruitless.

So, I agree with Mark here.  We should retry and fail bio-by-bio.  The
end effect wouldn't be much worse than block-by-block while taking much
less time.

-- 
tejun

Re: [PATCH] libata: reduce ATA command timeout to 7secs

[Alan]

> So, I agree with Mark here.  We should retry and fail bio-by-bio.  The
> end effect wouldn't be much worse than block-by-block while taking much
> less time.

And perhaps flip the bio list backwards as well 8) ?

Re: [PATCH] libata: reduce ATA command timeout to 7secs

[Mark Lord]

Alan wrote:
>> So, I agree with Mark here.  We should retry and fail bio-by-bio.  The
>> end effect wouldn't be much worse than block-by-block while taking much
>> less time.
> 
> And perhaps flip the bio list backwards as well 8) ?

Reversing I/O lists might be the thing to do, but it gets really
complex at the low level.  Drives don't have commands to read
sectors in reverse order.  So to reverse the I/O, each single
command would have to be broken out into possibly thousands
of individual reads..

Not something I'd want to implement, but that's why we have Tejun! ;)

Re: [PATCH] libata: reduce ATA command timeout to 7secs

[Mark Lord]

Tejun Heo wrote:
> Both ATA and ATAPI devices used the default timeouts defined by SCSI
> high level driver.  For both disks and ODDs, it was 30secs, which was
> way too long for disks.  This patch makes most ATA commands time out
> after 7secs - the de facto ATA command timeout, while leaving ATAPI
> timeout at 30secs.

Good patch, LONG overdue.

But 7 seconds is too short.  I have drives here now that
take slightly more than 7 seconds to report media errors.

With this patch, libata will timeout/reset the drive just before
it had a chance to tell us why it was taking too long.

The result would be that we might lose data unnecessarily,
especially in combination with the current SCSI policy of
blindly failing the entire request when only one sector was bad.

I've used 10 second timeouts with 100% success over the past 14 years,
which allows just enough margin (+2) for drives to finish their internal
retries and report back.

NACK to 7, ACK to 10 seconds.

Re: [PATCH] libata: reduce ATA command timeout to 7secs

[Tejun Heo]

[cc'ing Jens, hello]

Mark Lord wrote:
> Tejun Heo wrote:
>> Both ATA and ATAPI devices used the default timeouts defined by SCSI
>> high level driver.  For both disks and ODDs, it was 30secs, which was
>> way too long for disks.  This patch makes most ATA commands time out
>> after 7secs - the de facto ATA command timeout, while leaving ATAPI
>> timeout at 30secs.
> 
> Good patch, LONG overdue.
> 
> But 7 seconds is too short.  I have drives here now that
> take slightly more than 7 seconds to report media errors.
> 
> With this patch, libata will timeout/reset the drive just before
> it had a chance to tell us why it was taking too long.
> 
> The result would be that we might lose data unnecessarily,
> especially in combination with the current SCSI policy of
> blindly failing the entire request when only one sector was bad.

Timeout doesn't indicate media error, so SCSI will retry the command
several times.  So, those two don't interact.

> I've used 10 second timeouts with 100% success over the past 14 years,
> which allows just enough margin (+2) for drives to finish their internal
> retries and report back.
> 
> NACK to 7, ACK to 10 seconds.

Hmmm... I don't care whether 7 or 10.  The biggest reason I chose 7 was
because somebody told me that that's the de-facto default ATA command
timeout && the other os is using it.  I faintly remembers that it was
yourself or Jens.  Or I might be just imagining things again.  :-)

Jens, what do you think about this?

Thanks.

-- 
tejun