unreadable drives can be synchronized?

unreadable drives can be synchronized?

[Colin McCabe]

Hi all,

I am running software RAID on Linux 2.6.21.

While experimenting with adding and removing devices from the RAID array, I
noticed something very troubling. I have a bad drive (let's call it drive B)
which gets random read errors. I also have a good drive, call it drive A.

B can synchronize with A. But then, if I remove A from the raid array, A
cannot be re-added. This is because the bad drive, B, cannot be read from.

Basically, B appears to be "write-only"; it will never return an error on a
write, but just try to read from it, and you will be sorry.

Writing is fine:
[root@cmccabe-devel root]# dd if=/dev/zero of=/dev/sdb bs=524288
dd: writing `/dev/sdb': No space left on device
114464+0 records in
114463+0 records out

Reading is not:
[root@cmccabe-devel root]# dd if=/dev/sdb of=/dev/null bs=524288
ata1.00: exception Emask 0x0 SAct 0x3 SErr 0x0 action 0x2 frozen
ata1.00: cmd 60/00:00:00:b0:01/01:00:00:00:00/40 tag 0 cdb 0x0 data 131072 in
[ ... copious errors ... ]

I have disabled write caching using hdparm -W0.
Both drives are: Fujitsu MHV2060BH, 60 GB, Serial ATA
The SATA controller is: ICH6

My problem is that even though B gets into the synchronized state, it is no
good at all. This is potentially misleading, and if someone removes A after
synchronizing B, the system will probably crash, since there will be no good
drives left.

I wonder if anyone else is interested in a "paranoid recovery" mode where the
md layer tests the data that has been written. Even if this doubles the
recovery time, I think that it would be desirable for many applications.

Colin

Re: unreadable drives can be synchronized?

[Bill Davidsen]

Colin McCabe wrote:
> Hi all,
>
> I am running software RAID on Linux 2.6.21.
>
> While experimenting with adding and removing devices from the RAID 
> array, I
> noticed something very troubling. I have a bad drive (let's call it 
> drive B)
> which gets random read errors. I also have a good drive, call it drive A.
>
> B can synchronize with A. But then, if I remove A from the raid array, A
> cannot be re-added. This is because the bad drive, B, cannot be read 
> from.
>
> Basically, B appears to be "write-only"; it will never return an error 
> on a
> write, but just try to read from it, and you will be sorry.
>
You may be able to recover from this (why would you do such a thing?) by 
stopping the array and reassembling the array with only the "good" drive 
and the other as failed. Caution, I made this up, it should work but I 
have no bad drive to use for a test, we have a good recycling system in 
my area.
> Writing is fine:
> [root@cmccabe-devel root]# dd if=/dev/zero of=/dev/sdb bs=524288
> dd: writing `/dev/sdb': No space left on device
> 114464+0 records in
> 114463+0 records out
>
> Reading is not:
> [root@cmccabe-devel root]# dd if=/dev/sdb of=/dev/null bs=524288
> ata1.00: exception Emask 0x0 SAct 0x3 SErr 0x0 action 0x2 frozen
> ata1.00: cmd 60/00:00:00:b0:01/01:00:00:00:00/40 tag 0 cdb 0x0 data 
> 131072 in
> [ ... copious errors ... ]
>
> I have disabled write caching using hdparm -W0.
> Both drives are: Fujitsu MHV2060BH, 60 GB, Serial ATA
> The SATA controller is: ICH6
>
> My problem is that even though B gets into the synchronized state, it 
> is no
> good at all. This is potentially misleading, and if someone removes A 
> after
> synchronizing B, the system will probably crash, since there will be 
> no good
> drives left.
>
> I wonder if anyone else is interested in a "paranoid recovery" mode 
> where the
> md layer tests the data that has been written. Even if this doubles the
> recovery time, I think that it would be desirable for many applications.


-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: unreadable drives can be synchronized?

[Colin McCabe]

On 5/16/07, Bill Davidsen <davidsen@tmr.com> wrote:
> Colin McCabe wrote:
> > Hi all,
> >
> > I am running software RAID on Linux 2.6.21.
> >
> > While experimenting with adding and removing devices from the RAID
> > array, I
> > noticed something very troubling. I have a bad drive (let's call it
> > drive B)
> > which gets random read errors. I also have a good drive, call it drive A.
> >
> > B can synchronize with A. But then, if I remove A from the raid array, A
> > cannot be re-added. This is because the bad drive, B, cannot be read
> > from.
> >
> > Basically, B appears to be "write-only"; it will never return an error
> > on a
> > write, but just try to read from it, and you will be sorry.
> >
> You may be able to recover from this (why would you do such a thing?) by
> stopping the array and reassembling the array with only the "good" drive
> and the other as failed. Caution, I made this up, it should work but I
> have no bad drive to use for a test, we have a good recycling system in
> my area.

This is an embedded systems application. There isn't any important
data on drives A or B at the moment.

What concerns me is that apparently these Hitachi disks have errors
that only show up when you try to read from them. I don't know if this
is a firmware bug or a physical limitation of the way the drive
detects errors. I actually have two different drives which could fill
the role of drive B in this scenario.

If I do a "check" on both drives, it speedily removes B once it
realizes that it can't read from it. But what bothers me is that it is
able to become active without ever being tested by being read from. So
it seems like at minimum, careful admins should do a "check"
immediately after adding a new disk to an array.

Colin


> > Writing is fine:
> > [root@cmccabe-devel root]# dd if=/dev/zero of=/dev/sdb bs=524288
> > dd: writing `/dev/sdb': No space left on device
> > 114464+0 records in
> > 114463+0 records out
> >
> > Reading is not:
> > [root@cmccabe-devel root]# dd if=/dev/sdb of=/dev/null bs=524288
> > ata1.00: exception Emask 0x0 SAct 0x3 SErr 0x0 action 0x2 frozen
> > ata1.00: cmd 60/00:00:00:b0:01/01:00:00:00:00/40 tag 0 cdb 0x0 data
> > 131072 in
> > [ ... copious errors ... ]
> >
> > I have disabled write caching using hdparm -W0.
> > Both drives are: Fujitsu MHV2060BH, 60 GB, Serial ATA
> > The SATA controller is: ICH6
> >
> > My problem is that even though B gets into the synchronized state, it
> > is no
> > good at all. This is potentially misleading, and if someone removes A
> > after
> > synchronizing B, the system will probably crash, since there will be
> > no good
> > drives left.
> >
> > I wonder if anyone else is interested in a "paranoid recovery" mode
> > where the
> > md layer tests the data that has been written. Even if this doubles the
> > recovery time, I think that it would be desirable for many applications.
>
>
> --
> bill davidsen <davidsen@tmr.com>
>   CTO TMR Associates, Inc
>   Doing interesting things with small computers since 1979
>
>

Re: unreadable drives can be synchronized?

[Colin McCabe]

On 5/16/07, Colin McCabe <colin.p.mccabe@gmail.com> wrote:

> What concerns me is that apparently these Hitachi disks have errors

Sorry, I meant to write "Fujitsu," not "Hitachi."
Doh!

Colin

Re: unreadable drives can be synchronized?

[Neil Brown]

On Wednesday May 16, colin.p.mccabe@gmail.com wrote:
> Hi all,
> 
> I am running software RAID on Linux 2.6.21.
> 
> While experimenting with adding and removing devices from the RAID array, I
> noticed something very troubling. I have a bad drive (let's call it drive B)
> which gets random read errors. I also have a good drive, call it drive A.
> 
> B can synchronize with A. But then, if I remove A from the raid array, A
> cannot be re-added. This is because the bad drive, B, cannot be read from.

Well, if you remove A, then you have the situation of trusting your
data to a single drive.  And we all know that isn't very safe.

> 
> I wonder if anyone else is interested in a "paranoid recovery" mode where the
> md layer tests the data that has been written. Even if this doubles the
> recovery time, I think that it would be desirable for many applications.

I think it is just as easy to run a 'check' pass after recovery has
completed.

Whenever you are commissioning new hardware, it makes sense to do some
testing before you trust it with valuable data.
It sounds like this particular hardware failure lends itself to easy
discovery with a bit of testing - indeed, you found it while testing
your hardware.  So I don't think it is a failure mode we need to put
any extra care in to.
It is the failure modes that are hard to find with basic testing that
we should worry about.

NeilBrown

Re: unreadable drives can be synchronized?

[Andrew Burgess]

>Basically, B appears to be "write-only"; it will never return an error on a
>write, but just try to read from it, and you will be sorry.

It would be interesting to see what SMART says about drive B, especially
the short and long self tests.

Re: unreadable drives can be synchronized?

[Tomasz Chmielewski]

Andrew Burgess schrieb:
>> Basically, B appears to be "write-only"; it will never return an error on a
>> write, but just try to read from it, and you will be sorry.
> 
> It would be interesting to see what SMART says about drive B, especially
> the short and long self tests.

I wouldn't rely on SMART.

I have a broken drive, which has lots of badblocks - but SMART happily 
claims it's fine (short/long tests are completed without errors).


-- 
Tomasz Chmielewski
http://wpkg.org

Re: unreadable drives can be synchronized?

[Colin McCabe]

On 5/18/07, Tomasz Chmielewski <mangoo@wpkg.org> wrote:
> Andrew Burgess schrieb:
> >> Basically, B appears to be "write-only"; it will never return an error on a
> >> write, but just try to read from it, and you will be sorry.
> >
> > It would be interesting to see what SMART says about drive B, especially
> > the short and long self tests.
>
> I wouldn't rely on SMART.
>
> I have a broken drive, which has lots of badblocks - but SMART happily
> claims it's fine (short/long tests are completed without errors).
>

If you haven't seen Google's hard drive study yet, you should take a look.
It's at http://labs.google.com/papers/disk_failures.pdf

The conclusion says that "some of the SMART parameters are
well-correlated with higher failure probabilities," but also that "a
large fraction of [google's] failed drives have shown no SMART error
signals whatsoever."

Colin

Re: unreadable drives can be synchronized?

[Bill Davidsen]

Colin McCabe wrote:
> On 5/18/07, Tomasz Chmielewski <mangoo@wpkg.org> wrote:
>> Andrew Burgess schrieb:
>> >> Basically, B appears to be "write-only"; it will never return an 
>> error on a
>> >> write, but just try to read from it, and you will be sorry.
>> >
>> > It would be interesting to see what SMART says about drive B, 
>> especially
>> > the short and long self tests.
>>
>> I wouldn't rely on SMART.
>>
>> I have a broken drive, which has lots of badblocks - but SMART happily
>> claims it's fine (short/long tests are completed without errors).
>>
>
> If you haven't seen Google's hard drive study yet, you should take a 
> look.
> It's at http://labs.google.com/papers/disk_failures.pdf
>
> The conclusion says that "some of the SMART parameters are
> well-correlated with higher failure probabilities," but also that "a
> large fraction of [google's] failed drives have shown no SMART error
> signals whatsoever."
Having covered that in a presentation to a user group related to SMART. 
may I offer a paraphrase which may be more obvious to people who are not 
native speakers of English:

High counts of some SMART parameters indicate that the drive is likely 
to fail. However, most drives fail without warning.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: unreadable drives can be synchronized?

[Colin McCabe]

Andrew Burgess wrote:
>> Basically, B appears to be "write-only"; it will never return an error on a
>> write, but just try to read from it, and you will be sorry.
> 
> It would be interesting to see what SMART says about drive B, especially
> the short and long self tests.
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

On my hard drives, automatic online testing is turned on, and so is 
automatic offline testing. I run the long self-test once a week.

I have two drives which can play the role of B. One of them has this 
SMART output:

[root@cmccabe-devel root]# smartctl -d ata /dev/sdb -H
smartctl version 5.36 [i686-pc-linux-gnu] Copyright (C) 2002-6 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: FAILED!
Drive failure expected in less than 24 hours. SAVE ALL DATA.
Failed Attributes:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE 
UPDATED  WHEN_FAILED RAW_VALUE
   5 Reallocated_Sector_Ct   0x0033   004   004   024    Pre-fail 
Always   FAILING_NOW 133273031255

The other one passes SMART.
Both of them eat data, though.

Colin