Driver retries disk errors.

Driver retries disk errors.

[Rogier Wolff]

Hi, 

We encounter "bad" drives with quite a lot more regularity than other
people (look at the Email address). We're however, wondering why the
IDE code still retries a bad block 8 times? By the time the drive
reports "bad block" it has already tried it several times, including a
bunch of "recalibrates" etc etc. For comparison, the Scsi-disk driver
doesn't do any retrying.

So, why do we still do this?

- The driver may still work for MFM drives and less "intelligent"
  controllers?

- Someone has recently seen that this actually helps?


In fact we regularly are able to recover data from drives: we have a
userspace application that retries over and over again, and this
sometimes recovers "marginal" blocks. This could be considered "good
practise" if there is a filesystem requesting the block. On the other
hand, when this happens, the drive is usually beyond being usable for
a filesystem: if we recover one block this way, the next block will be
errorred and the filesystem "crashes" anyway. In fact this behaviour
may masquerade the first warnings that something is going wrong....

So, I'm arguing for: We remove the retry code alltogether, OR we make
an option to re-enable the retry code for MFM era drives(*) (Note: those 
are more than 10 years old, so almost (but not quite) extinct).

	Roger. 

(*) Note: Tested last month: The driver still works for MFM
drives. However, the initialization apparently is not enough
anymore. The drive did not work when the BIOS didn't think there was a
drive.

-- 
+-- Rogier Wolff -- www.harddisk-recovery.nl -- 0800 220 20 20 --
| Files foetsie, bestanden kwijt, alle data weg?!
| Blijf kalm en neem contact op met Harddisk-recovery.nl!

Re: Driver retries disk errors.

[Theodore Ts'o]

On Mon, Aug 30, 2004 at 06:39:31PM +0200, Rogier Wolff wrote:
> We encounter "bad" drives with quite a lot more regularity than other
> people (look at the Email address). We're however, wondering why the
> IDE code still retries a bad block 8 times? 

I could see retrying 2 or 3 times, but 8 times does seem to be a bit
much, agreed.

> In fact we regularly are able to recover data from drives: we have a
> userspace application that retries over and over again, and this
> sometimes recovers "marginal" blocks. This could be considered "good
> practise" if there is a filesystem requesting the block. On the other
> hand, when this happens, the drive is usually beyond being usable for
> a filesystem: if we recover one block this way, the next block will be
> errorred and the filesystem "crashes" anyway. In fact this behaviour
> may masquerade the first warnings that something is going wrong....

If the block gets successfully read after 2 or 3 tries, it might be a
good idea for the kernel to automatically do a forced rewrite of the
block, which should cause the disk to do its own disk block
sparing/reassignment.  

						- Ted

Re: Driver retries disk errors.

[James Courtier-Dutton]

Theodore Ts'o wrote:
> On Mon, Aug 30, 2004 at 06:39:31PM +0200, Rogier Wolff wrote:
> 
>>We encounter "bad" drives with quite a lot more regularity than other
>>people (look at the Email address). We're however, wondering why the
>>IDE code still retries a bad block 8 times? 
> 
> 
> I could see retrying 2 or 3 times, but 8 times does seem to be a bit
> much, agreed.
> 
> 
>>In fact we regularly are able to recover data from drives: we have a
>>userspace application that retries over and over again, and this
>>sometimes recovers "marginal" blocks. This could be considered "good
>>practise" if there is a filesystem requesting the block. On the other
>>hand, when this happens, the drive is usually beyond being usable for
>>a filesystem: if we recover one block this way, the next block will be
>>errorred and the filesystem "crashes" anyway. In fact this behaviour
>>may masquerade the first warnings that something is going wrong....
> 
> 
> If the block gets successfully read after 2 or 3 tries, it might be a
> good idea for the kernel to automatically do a forced rewrite of the
> block, which should cause the disk to do its own disk block
> sparing/reassignment.  
> 
> 						- Ted

It does the same retries with CD-ROM and DVDs, and if the retries fail, 
it disables DMA! It even does the retries when reading CD-Audio.
Maybe there should be a "retrys" setting that can be set by hdparm, then 
we could set the retry counts, and what happens when a retry fails on a 
per device basis.

Re: Driver retries disk errors.

[Rogier Wolff]

On Mon, Aug 30, 2004 at 07:26:27PM +0100, James Courtier-Dutton wrote:
> It does the same retries with CD-ROM and DVDs, and if the retries fail, 
> it disables DMA!

As a matter of fact, we've had a computer where I tried to
get an MFM drive working. There I had changed lots of settings
in the BIOS to disable the onboard IDE and stuff like that. 
When we tried to get IDE back working, we encountered the 
situation where the secondary channel would not DMA unless
<something in the BIOS>. There the strategy "disable DMA"
works: the drive is "switched down" and something works. 

I remember from the old days that this was: "To enable
the user to continue to use the system to fix the problem". 

However, in practise, this failure is not something that
you can fix "if you have access to your drive", but something
you get to fix in the BIOS. So does this still help? 

Well, maybe PIO is so "basic" that it will always work, 
and is a good "last resort'.

The "same retries with CDROM" stems from the fact that the
code was initially duplicated. (as in cp ide-disk.c ide-cd.c) 
The fact that DVDs are nowadays writable should be a hint that 
the drivers may be better off getting merged one of these days. 

	Roger. 

-- 
+-- Rogier Wolff -- www.harddisk-recovery.nl -- 0800 220 20 20 --
| Files foetsie, bestanden kwijt, alle data weg?!
| Blijf kalm en neem contact op met Harddisk-recovery.nl!

Re: Driver retries disk errors.

[Alan Cox]

On Llu, 2004-08-30 at 19:26, James Courtier-Dutton wrote:
> Theodore Ts'o wrote:
> > If the block gets successfully read after 2 or 3 tries, it might be a
> > good idea for the kernel to automatically do a forced rewrite of the
> > block, which should cause the disk to do its own disk block
> > sparing/reassignment.  

Not really as far as I can tell. It isn't a disk any more, its a storage
appliance on a funny connector. It already knows a lot about retries
internally as well as rewriting blocks with high ECC error
count. In fact you actually have to issue a different command to do 
read/write without retry.

> It does the same retries with CD-ROM and DVDs, and if the retries fail, 
> it disables DMA! It even does the retries when reading CD-Audio.
> Maybe there should be a "retrys" setting that can be set by hdparm, then 
> we could set the retry counts, and what happens when a retry fails on a 
> per device basis.

It probably should be smarter about error strategy here. You can use
hdparm to control some of this in the IDE case but not enough.

Alan

Re: Driver retries disk errors.

[Eric Mudama]

On Tue, 31 Aug 2004 12:38:45 +0100, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
> Not really as far as I can tell. It isn't a disk any more, its a storage
> appliance on a funny connector. It already knows a lot about retries
> internally as well as rewriting blocks with high ECC error
> count. In fact you actually have to issue a different command to do
> read/write without retry.

True, but in the later versions of the ATA specification, the retry
option was depreciated.  I think you'll find virtually every ATA drive
built today ignores that "suggestion" from the host.

--eric

Re: Driver retries disk errors.

[Rogier Wolff]

On Mon, Aug 30, 2004 at 01:46:32PM -0400, Theodore Ts'o wrote:
> > a filesystem: if we recover one block this way, the next block will be
> > errorred and the filesystem "crashes" anyway. In fact this behaviour
> > may masquerade the first warnings that something is going wrong....
> 
> If the block gets successfully read after 2 or 3 tries, it might be a
> good idea for the kernel to automatically do a forced rewrite of the
> block, which should cause the disk to do its own disk block
> sparing/reassignment.  

Hi Ted, 

I agree that this is the theory. In practise however, I've never
seen it work correctly. We've seen several disks with say 1-5 bad
blocks and nothing else, and "dd if=/dev/zero of=/dev/<disk>" doesn't
seem to cure them.

	Roger. 

-- 
+-- Rogier Wolff -- www.harddisk-recovery.nl -- 0800 220 20 20 --
| Files foetsie, bestanden kwijt, alle data weg?!
| Blijf kalm en neem contact op met Harddisk-recovery.nl!

Re: Driver retries disk errors.

[Alan Cox]

On Llu, 2004-08-30 at 17:39, Rogier Wolff wrote:
> We encounter "bad" drives with quite a lot more regularity than other
> people (look at the Email address). We're however, wondering why the
> IDE code still retries a bad block 8 times? By the time the drive
> reports "bad block" it has already tried it several times, including a
> bunch of "recalibrates" etc etc. For comparison, the Scsi-disk driver
> doesn't do any retrying.

It helps for some things like magneto-opticals. For generic hard drives
its only relevant for older devices.

> (*) Note: Tested last month: The driver still works for MFM
> drives. However, the initialization apparently is not enough
> anymore. The drive did not work when the BIOS didn't think there was a
> drive.

Please file a bug report if 2.6 also shows that problem.

Re: Driver retries disk errors.

[Andre Hedrick]

Rogier,

Because the command layer states to execute retries, regardless.
Modern drives now convert read-once to retry.
You need special opcodes to revert to desired status.

Media forensics is not a cake walk.

Andre Hedrick
LAD Storage Consulting Group

On Tue, 31 Aug 2004, Alan Cox wrote:

> On Llu, 2004-08-30 at 17:39, Rogier Wolff wrote:
> > We encounter "bad" drives with quite a lot more regularity than other
> > people (look at the Email address). We're however, wondering why the
> > IDE code still retries a bad block 8 times? By the time the drive
> > reports "bad block" it has already tried it several times, including a
> > bunch of "recalibrates" etc etc. For comparison, the Scsi-disk driver
> > doesn't do any retrying.
> 
> It helps for some things like magneto-opticals. For generic hard drives
> its only relevant for older devices.
> 
> > (*) Note: Tested last month: The driver still works for MFM
> > drives. However, the initialization apparently is not enough
> > anymore. The drive did not work when the BIOS didn't think there was a
> > drive.
> 
> Please file a bug report if 2.6 also shows that problem.
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: Driver retries disk errors.

[Rogier Wolff]

On Tue, Aug 31, 2004 at 12:45:15PM +0100, Alan Cox wrote:
> On Llu, 2004-08-30 at 17:39, Rogier Wolff wrote:
> > We encounter "bad" drives with quite a lot more regularity than other
> > people (look at the Email address). We're however, wondering why the
> > IDE code still retries a bad block 8 times? By the time the drive
> > reports "bad block" it has already tried it several times, including a
> > bunch of "recalibrates" etc etc. For comparison, the Scsi-disk driver
> > doesn't do any retrying.
> 
> It helps for some things like magneto-opticals. For generic hard drives
> its only relevant for older devices.
> 
> > (*) Note: Tested last month: The driver still works for MFM
> > drives. However, the initialization apparently is not enough
> > anymore. The drive did not work when the BIOS didn't think there was a
> > drive.
> 
> Please file a bug report if 2.6 also shows that problem.

Will try to test when we have time. 

So, can we agree on: 
	- might be needed for 
		- Floppies?
		- MO drives
		- older drives

Can we auto-detect these cases (Linus doesn't like configurable
parameters that need tweaking to work well, and I agree: 99% of the
users want to have stuff that works (well) out of the box.)


How about we set the num-retries to 1, and increase to 8 for
"weird devices" (floppy, MO), and older drives. 

How do we detect: "Older drives"? Would "MFM": the user specified the
geometry" be valid as a detection of "older drive"?

Or do we want to include the 40Mb-1G generation drives as well? How
do we detect those if we want to include them? 

I do want to make the num_retries thing a configurable parameter,
should the autodetect get it wrong: We get drives that we want to
recover without the kernel-level retries...

(still: I argue that you need to consider a "retry-works" error as an
early warning that your media is going bad, and you need to get your
data off ASAP! If the kernel silently retries and succeeds, the user
won't notice a thing and continue using the drive (or MO media) until
the error becomes irrecoverable. I recommend we put the retry at the
user level. As in "person behind keyboard".)

I'll try to make a patch as long as we work towards a feature set
first.... 


	Roger. 

-- 
+-- Rogier Wolff -- www.harddisk-recovery.nl -- 0800 220 20 20 --
| Files foetsie, bestanden kwijt, alle data weg?!
| Blijf kalm en neem contact op met Harddisk-recovery.nl!

Re: Driver retries disk errors.

[Alan Cox]

On Maw, 2004-08-31 at 14:54, Rogier Wolff wrote:
> So, can we agree on: 
> 	- might be needed for 
> 		- Floppies?
> 		- MO drives
> 		- older drives

Other random stuff it saves our backside on we don't know about.

> How about we set the num-retries to 1, and increase to 8 for
> "weird devices" (floppy, MO), and older drives. 

Disagree. I want it robust. If you want to set low retry counts then
the user should do so for special cases like forensics.

> I do want to make the num_retries thing a configurable parameter,
> should the autodetect get it wrong: We get drives that we want to
> recover without the kernel-level retries...

Making it configurable is good, but I can't help feeling that this
belongs at the block layer - I wonder what Jens thinks, it might well
have to be done by the driver because only the driver knows enough but
the ioctl/config option ought to be common.

> (still: I argue that you need to consider a "retry-works" error as an
> early warning that your media is going bad, and you need to get your
> data off ASAP! If the kernel silently retries and succeeds, the user
> won't notice a thing and continue using the drive (or MO media) until
> the error becomes irrecoverable. I recommend we put the retry at the
> user level. As in "person behind keyboard".)

M/O media retries generally do the right thing and have the right
effect. If you want to know if your drive is failing use SMART and ask
the drive

Remember: Storage appliance not disk. Treat it like a storage
appliance and you'll get better results.

Alan

Re: Driver retries disk errors.

[Erik Mouw]

On Tue, Aug 31, 2004 at 03:12:52PM +0100, Alan Cox wrote:
> On Maw, 2004-08-31 at 14:54, Rogier Wolff wrote:
> > How about we set the num-retries to 1, and increase to 8 for
> > "weird devices" (floppy, MO), and older drives. 
> 
> Disagree. I want it robust. If you want to set low retry counts then
> the user should do so for special cases like forensics.

The SCSI disk driver has been doing a single retry for quite some time
and it hasn't really bitten people. Why would the IDE disk driver be
different? The only case I can imagine a retry would be OK, is when we
get an UDMA CRC error (caused by bad cables).

(OK, for SCSI drives you have a lot more control about how a drive
should treat errors, but the kernel will not retry a block when the
drive reported it's bad.)


Erik

-- 
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands

Re: Driver retries disk errors.

[Alan Cox]

On Maw, 2004-08-31 at 16:56, Erik Mouw wrote:
> The SCSI disk driver has been doing a single retry for quite some time
> and it hasn't really bitten people. Why would the IDE disk driver be
> different? The only case I can imagine a retry would be OK, is when we
> get an UDMA CRC error (caused by bad cables).

Retries also pop up in other less obvious cases and conveniently paper
over a wide variety of timeouts, power management quirks and drives just
having a random fit. Eight is probably excessive in all cases.

For non hard disk cases many devices do want and need retry.

Alan

Re: Driver retries disk errors.

[Erik Mouw]

On Tue, Aug 31, 2004 at 04:13:54PM +0100, Alan Cox wrote:
> On Maw, 2004-08-31 at 16:56, Erik Mouw wrote:
> > The SCSI disk driver has been doing a single retry for quite some time
> > and it hasn't really bitten people. Why would the IDE disk driver be
> > different? The only case I can imagine a retry would be OK, is when we
> > get an UDMA CRC error (caused by bad cables).
> 
> Retries also pop up in other less obvious cases and conveniently paper
> over a wide variety of timeouts, power management quirks and drives just
> having a random fit. Eight is probably excessive in all cases.

There are indeed all sorts of other retries in various layers, the
worst one when the kernel tries to read-ahead a couple of blocks on an
IDE disk(1). You can work around them with raw IO or O_DIRECT.

> For non hard disk cases many devices do want and need retry.

And many others do not. CompactFlash readers are usually implemented as
a USB storage device, which on its turn is implemented as a SCSI
"disk". So far I haven't seen a CompactFlash which could be "fixed" by
retries.

iSCSI appliances can also make things worse: when the target machine is
implemented as a simple "pass everything to the real SCSI disk" device,
it's not really different from a directly connected SCSI disk. However,
when the iSCSI target interprets the SCSI commands and has its own way
to deal with bad blocks (i.e.: it retries the blocks), things can get
very bad when the kernel also uses a couple of retries.

In my experience it would be good if the IDE disk driver would behave
like the SCSI disk driver: no retries on a bad block. I agree that it
would be a good idea to make it configurable through the block layer to
avoid code duplication (blockdev --getretries/--setretries).


Erik

(1) Imagine an application doing a linear read on a file with an 8
block read ahead and the last block being bad. The kernel will try to
read that bad block 16 times, but because the IDE driver also has 8
retries, the kernel will try to read that bad block *64* times. It
usually takes an IDE drive about 2 seconds to figure out a block is
bad, so the application gets stuck for 2 minutes in that single bad
block.

-- 
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands

Re: Driver retries disk errors.

[Alan Cox]

On Maw, 2004-08-31 at 18:00, Erik Mouw wrote:
> > For non hard disk cases many devices do want and need retry.
> 
> And many others do not. CompactFlash readers are usually implemented as
> a USB storage device, which on its turn is implemented as a SCSI
> "disk". So far I haven't seen a CompactFlash which could be "fixed" by
> retries.

It does no harm trying. It does real harm not being conservative and
losing peoples data. You recover people's data after its lost, the
IDE layer's job is to make sure it doesn't get lost in the first place.

> (1) Imagine an application doing a linear read on a file with an 8
> block read ahead and the last block being bad. The kernel will try to
> read that bad block 16 times, but because the IDE driver also has 8
> retries, the kernel will try to read that bad block *64* times. It
> usually takes an IDE drive about 2 seconds to figure out a block is
> bad, so the application gets stuck for 2 minutes in that single bad
> block.

Right now I know of no way to tell which is readahead for a failed
command or of telling the block layer to forget them. Fix this at the
block layer and IDE can abort the readahead sequence happily enough
because IDE is too dumb to have issued further commands to the drive at
this point.

Alan

Re: Driver retries disk errors.

[Christer Weinigel]

Alan Cox <alan@lxorguk.ukuu.org.uk> writes:

> Retries also pop up in other less obvious cases and conveniently paper
> over a wide variety of timeouts, power management quirks and drives just
> having a random fit. Eight is probably excessive in all cases.
> 
> For non hard disk cases many devices do want and need retry.

For ripping CDs I'd prefer if the application could control retries
and not default to what the CD player prefers.

I have a CD, KLF -- Ultra Rare Tracks, and unfortunately someone I
borrowed it to managed to literally stomp on it, so the record has a
lot of scratches.  I would very much like to rip as much as possible
of this CD so that I can at least listen to part of it, but every time
I have tried the CD player just gets stuck in an endless retry loop.

The same thing happens when trying to rip a "copy controlled" CD,
which is also pretty irritating since this means that I can't rip my
Teddybears Stockholm CD and put the songs on my iPod.

  /Christer

-- 
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <christer@weinigel.se>  http://www.weinigel.se