Multiple disk failure, but slot numbers are corrupt and preventing assembly.

Multiple disk failure, but slot numbers are corrupt and preventing assembly.

[Leon Woestenberg]

Hello,

it's recovery time again. Problem at hand: raid5 consisting of four
partitions, each on a drive. Two disks have failed. Assembly fails
because the slot numbers of the array components seem to be corrupt.

/dev/md0 consisting of /dev/sd[abcd]1, of which b,c failed and of
which c seems really bad in SMART, b looks reasonably OK judging from
SMART.

Checksum of the failed component superblocks was bad.

Using mdadm.conf we have already tried updating the superblocks. This
partly succeeded in the sense that checksums came up ok, the slot
numbers did not.

mdadm refuses to assemble, even with --force.

Could you guys peek over the array configuration (mdadm --examine) and
see if there is a non-destructive way to try and mount the array. If
not, what is the least intrusive way to do a non-syncing (re)create?

Data recovery is our prime concern here.

Below the uname -a, --examine output of all four drives, mdadm.conf of
what we think the array should look like and finally, the mdadm
--assemble command and output.

Note the slot numbers on /dev/sd[bc].

Thanks for any help,

with kind regards,

Leon Woestenberg




Linux localhost 2.6.16.14-axon1 #1 SMP PREEMPT Mon May 8 17:01:33 CEST
2006 i486 pentium4 i386 GNU/Linux

[root@localhost ~]# mdadm --examine /dev/sda1
/dev/sda1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 51a95144:00af4c77:c1cd173b:94cb1446
  Creation Time : Mon Sep  5 13:16:42 2005
     Raid Level : raid5
    Device Size : 390620352 (372.52 GiB 400.00 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0

    Update Time : Tue Apr 17 07:03:46 2007
          State : active
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0
       Checksum : f98ed71b - correct
         Events : 0.115909229

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     0       8        1        0      active sync   /dev/sda1

   0     0       8        1        0      active sync   /dev/sda1
   1     1       8       17        1      active sync   /dev/sdb1
   2     2       8       33        2      active sync   /dev/sdc1
   3     3       8       49        3      active sync   /dev/sdd1
[root@localhost ~]# mdadm --examine /dev/sdb1
/dev/sdb1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 51a95144:00af4c77:c1cd173b:94cb1446
  Creation Time : Mon Sep  5 13:16:42 2005
     Raid Level : raid5
    Device Size : 390620352 (372.52 GiB 400.00 GB)
   Raid Devices : 4
  Total Devices : 5
Preferred Minor : 0

    Update Time : Tue Apr 17 07:03:46 2007
          State : clean
 Active Devices : 5
Working Devices : 4
 Failed Devices : 1
  Spare Devices : 0
       Checksum : e6d35288 - correct
         Events : 0.115909230

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this -11221199   -1288577935    -1551230943    2035285809      faulty
active removed

   0     0       8        1        0      active sync   /dev/sda1
   1     1       8       17        1      active sync   /dev/sdb1
   2     2       8       33        2      active sync   /dev/sdc1
   3     3       8       49        3      active sync   /dev/sdd1
[root@localhost ~]# mdadm --examine /dev/sdc1
/dev/sdc1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 51a95144:00af4c77:c1cd173b:94cb1446
  Creation Time : Mon Sep  5 13:16:42 2005
     Raid Level : raid5
    Device Size : 390620352 (372.52 GiB 400.00 GB)
   Raid Devices : 4
  Total Devices : 9
Preferred Minor : 0

    Update Time : Tue Apr 17 07:03:46 2007
          State : clean
 Active Devices : 8
Working Devices : 8
 Failed Devices : 1
  Spare Devices : 0
       Checksum : 33e911c - correct
         Events : 0.115909230

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this 1038288281   293191225    29538921    -2128142983      faulty
active write-mostly

   0     0       8        1        0      active sync   /dev/sda1
   1     1       8       17        1      active sync   /dev/sdb1
   2     2       8       33        2      active sync   /dev/sdc1
   3     3       8       49        3      active sync   /dev/sdd1
[root@localhost ~]# mdadm --examine /dev/sdd1
/dev/sdd1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 51a95144:00af4c77:c1cd173b:94cb1446
  Creation Time : Mon Sep  5 13:16:42 2005
     Raid Level : raid5
    Device Size : 390620352 (372.52 GiB 400.00 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0

    Update Time : Tue Apr 17 07:03:46 2007
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 7779c2 - correct
         Events : 0.115909230

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     3       8       49        3      active sync   /dev/sdd1

   0     0       8        1        0      active sync   /dev/sda1
   1     1       8       17        1      active sync   /dev/sdb1
   2     2       8       33        2      active sync   /dev/sdc1
   3     3       8       49        3      active sync   /dev/sdd1
[root@localhost ~]#

[root@localhost ~]# cat /tmp/mdadm.conf
DEVICE /dev/sda1 /dev/sdb1/ /dev/sdc1 /dev/sdd1
ARRAY /dev/md0 devices=/dev/sda1,/dev/sdb1,/dev/sdc1,/dev/sdd1

[root@localhost ~]# mdadm -v --assemble --scan --config=/tmp/mdadm.conf --force
mdadm: looking for devices for /dev/md0
mdadm: /dev/sda1 is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 2035285809.
mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot -2128142983.
mdadm: /dev/sdd1 is identified as a member of /dev/md0, slot 3.
mdadm: no uptodate device for slot 1 of /dev/md0
mdadm: no uptodate device for slot 2 of /dev/md0
mdadm: added /dev/sdd1 to /dev/md0 as 3
mdadm: added /dev/sda1 to /dev/md0 as 0
mdadm: /dev/md0 assembled from 2 drives - not enough to start the array.




-- 
Leon

Re: Multiple disk failure, but slot numbers are corrupt and preventing assembly.

[David Greaves]

There is some odd stuff in there:

/dev/sda1:
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Events : 0.115909229

/dev/sdb1:
Active Devices : 5
Working Devices : 4
Failed Devices : 1
Events : 0.115909230

/dev/sdc1:
Active Devices : 8
Working Devices : 8
Failed Devices : 1
Events : 0.115909230

/dev/sdd1:
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Events : 0.115909230

but your event counts are consistent. It looks like corruption on 2 disks :(
Or did you try some things?

I think you'll need to recreate the array since assemble can't figure things out.

Since you mention SMART errors on /dev/sdb you are taking a big chance by trying
to start up the array with a known faulty disk - especially if you resync as
it's a very IO intensive operation that will read every sector of the bad disk
and is likely to trigger errors that will kick it again leaving you back where
you started (or worse).

If you are desperate for data recovery and you have the space then you should
take disk images using ddrescue *before* trying anything.

Next best is if you are buying new disks and can wait for them to arrive, do so.
You can then use ddrescue to copy the old disk to the new ones and work with
non-broken hardware.

If you have no choice....

From this point forward it will be very easy to mess up.


Once you have disks to work on you can try to recreate the array.

You were using 0.9 superblocks, 64k, left symmetric which are defaults.

You should re-create in degraded mode to prevent the sync from starting (if you
got the order wrong then it would get the parity calc wrong).

So:
mdadm --create /dev/md0 --force -l5 -n4 /dev/sda1 /dev/sdb1 missing /dev/sdc1

Then do a *readonly* fsck on the /dev/md0.

If it works you can try a backup or an fsck.

Ask if anything isn't clear.

David
PS I recovered from a 2-disk failure last night. Seems to be back up and
re-syncing :) Glad I had a spare disk around!

Leon Woestenberg wrote:
> Hello,
> 
> it's recovery time again. Problem at hand: raid5 consisting of four
> partitions, each on a drive. Two disks have failed. Assembly fails
> because the slot numbers of the array components seem to be corrupt.
> 
> /dev/md0 consisting of /dev/sd[abcd]1, of which b,c failed and of
> which c seems really bad in SMART, b looks reasonably OK judging from
> SMART.
> 
> Checksum of the failed component superblocks was bad.
> 
> Using mdadm.conf we have already tried updating the superblocks. This
> partly succeeded in the sense that checksums came up ok, the slot
> numbers did not.
> 
> mdadm refuses to assemble, even with --force.
> 
> Could you guys peek over the array configuration (mdadm --examine) and
> see if there is a non-destructive way to try and mount the array. If
> not, what is the least intrusive way to do a non-syncing (re)create?
> 
> Data recovery is our prime concern here.
> 
> Below the uname -a, --examine output of all four drives, mdadm.conf of
> what we think the array should look like and finally, the mdadm
> --assemble command and output.
> 
> Note the slot numbers on /dev/sd[bc].
> 
> Thanks for any help,
> 
> with kind regards,
> 
> Leon Woestenberg
> 
> 
> 
> 
> Linux localhost 2.6.16.14-axon1 #1 SMP PREEMPT Mon May 8 17:01:33 CEST
> 2006 i486 pentium4 i386 GNU/Linux
> 
> [root@localhost ~]# mdadm --examine /dev/sda1
> /dev/sda1:
>          Magic : a92b4efc
>        Version : 00.90.00
>           UUID : 51a95144:00af4c77:c1cd173b:94cb1446
>  Creation Time : Mon Sep  5 13:16:42 2005
>     Raid Level : raid5
>    Device Size : 390620352 (372.52 GiB 400.00 GB)
>   Raid Devices : 4
>  Total Devices : 4
> Preferred Minor : 0
> 
>    Update Time : Tue Apr 17 07:03:46 2007
>          State : active
> Active Devices : 4
> Working Devices : 4
> Failed Devices : 0
>  Spare Devices : 0
>       Checksum : f98ed71b - correct
>         Events : 0.115909229
> 
>         Layout : left-symmetric
>     Chunk Size : 64K
> 
>      Number   Major   Minor   RaidDevice State
> this     0       8        1        0      active sync   /dev/sda1
> 
>   0     0       8        1        0      active sync   /dev/sda1
>   1     1       8       17        1      active sync   /dev/sdb1
>   2     2       8       33        2      active sync   /dev/sdc1
>   3     3       8       49        3      active sync   /dev/sdd1
> [root@localhost ~]# mdadm --examine /dev/sdb1
> /dev/sdb1:
>          Magic : a92b4efc
>        Version : 00.90.00
>           UUID : 51a95144:00af4c77:c1cd173b:94cb1446
>  Creation Time : Mon Sep  5 13:16:42 2005
>     Raid Level : raid5
>    Device Size : 390620352 (372.52 GiB 400.00 GB)
>   Raid Devices : 4
>  Total Devices : 5
> Preferred Minor : 0
> 
>    Update Time : Tue Apr 17 07:03:46 2007
>          State : clean
> Active Devices : 5
> Working Devices : 4
> Failed Devices : 1
>  Spare Devices : 0
>       Checksum : e6d35288 - correct
>         Events : 0.115909230
> 
>         Layout : left-symmetric
>     Chunk Size : 64K
> 
>      Number   Major   Minor   RaidDevice State
> this -11221199   -1288577935    -1551230943    2035285809      faulty
> active removed
> 
>   0     0       8        1        0      active sync   /dev/sda1
>   1     1       8       17        1      active sync   /dev/sdb1
>   2     2       8       33        2      active sync   /dev/sdc1
>   3     3       8       49        3      active sync   /dev/sdd1
> [root@localhost ~]# mdadm --examine /dev/sdc1
> /dev/sdc1:
>          Magic : a92b4efc
>        Version : 00.90.00
>           UUID : 51a95144:00af4c77:c1cd173b:94cb1446
>  Creation Time : Mon Sep  5 13:16:42 2005
>     Raid Level : raid5
>    Device Size : 390620352 (372.52 GiB 400.00 GB)
>   Raid Devices : 4
>  Total Devices : 9
> Preferred Minor : 0
> 
>    Update Time : Tue Apr 17 07:03:46 2007
>          State : clean
> Active Devices : 8
> Working Devices : 8
> Failed Devices : 1
>  Spare Devices : 0
>       Checksum : 33e911c - correct
>         Events : 0.115909230
> 
>         Layout : left-symmetric
>     Chunk Size : 64K
> 
>      Number   Major   Minor   RaidDevice State
> this 1038288281   293191225    29538921    -2128142983      faulty
> active write-mostly
> 
>   0     0       8        1        0      active sync   /dev/sda1
>   1     1       8       17        1      active sync   /dev/sdb1
>   2     2       8       33        2      active sync   /dev/sdc1
>   3     3       8       49        3      active sync   /dev/sdd1
> [root@localhost ~]# mdadm --examine /dev/sdd1
> /dev/sdd1:
>          Magic : a92b4efc
>        Version : 00.90.00
>           UUID : 51a95144:00af4c77:c1cd173b:94cb1446
>  Creation Time : Mon Sep  5 13:16:42 2005
>     Raid Level : raid5
>    Device Size : 390620352 (372.52 GiB 400.00 GB)
>   Raid Devices : 4
>  Total Devices : 4
> Preferred Minor : 0
> 
>    Update Time : Tue Apr 17 07:03:46 2007
>          State : clean
> Active Devices : 4
> Working Devices : 4
> Failed Devices : 0
>  Spare Devices : 0
>       Checksum : 7779c2 - correct
>         Events : 0.115909230
> 
>         Layout : left-symmetric
>     Chunk Size : 64K
> 
>      Number   Major   Minor   RaidDevice State
> this     3       8       49        3      active sync   /dev/sdd1
> 
>   0     0       8        1        0      active sync   /dev/sda1
>   1     1       8       17        1      active sync   /dev/sdb1
>   2     2       8       33        2      active sync   /dev/sdc1
>   3     3       8       49        3      active sync   /dev/sdd1
> [root@localhost ~]#
> 
> [root@localhost ~]# cat /tmp/mdadm.conf
> DEVICE /dev/sda1 /dev/sdb1/ /dev/sdc1 /dev/sdd1
> ARRAY /dev/md0 devices=/dev/sda1,/dev/sdb1,/dev/sdc1,/dev/sdd1
> 
> [root@localhost ~]# mdadm -v --assemble --scan --config=/tmp/mdadm.conf
> --force
> mdadm: looking for devices for /dev/md0
> mdadm: /dev/sda1 is identified as a member of /dev/md0, slot 0.
> mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 2035285809.
> mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot -2128142983.
> mdadm: /dev/sdd1 is identified as a member of /dev/md0, slot 3.
> mdadm: no uptodate device for slot 1 of /dev/md0
> mdadm: no uptodate device for slot 2 of /dev/md0
> mdadm: added /dev/sdd1 to /dev/md0 as 3
> mdadm: added /dev/sda1 to /dev/md0 as 0
> mdadm: /dev/md0 assembled from 2 drives - not enough to start the array.
> 
> 
> 
> 

Re: Multiple disk failure, but slot numbers are corrupt and preventing assembly.

[Leon Woestenberg]

Hello,

On 4/23/07, David Greaves <david@dgreaves.com> wrote:
> There is some odd stuff in there:
>
> /dev/sda1:
> Active Devices : 4
> Working Devices : 4
> Failed Devices : 0
> Events : 0.115909229
>
> /dev/sdb1:
> Active Devices : 5
> Working Devices : 4
> Failed Devices : 1
> Events : 0.115909230
>
> /dev/sdc1:
> Active Devices : 8
> Working Devices : 8
> Failed Devices : 1
> Events : 0.115909230
>
> /dev/sdd1:
> Active Devices : 4
> Working Devices : 4
> Failed Devices : 0
> Events : 0.115909230
>
> but your event counts are consistent. It looks like corruption on 2 disks :(

Exactly.

> Or did you try some things?
>
We tried updating the superblocks. It did not help, there remains
corrupt data somehow:

[root@localhost ~]# mdadm --examine /dev/sdb1
/dev/sdb1:
[...]
     Number   Major   Minor   RaidDevice State
this -11221199   -1288577935    -1551230943    2035285809      faulty
active removed
[...]

[root@localhost ~]# mdadm --examine /dev/sdc1
/dev/sdc1:
[...]
     Number   Major   Minor   RaidDevice State
this 1038288281   293191225    29538921    -2128142983      faulty
active write-mostly
[...]

That seems exactly what mdadm barfs on:

[root@localhost ~]# mdadm -v --assemble --scan --config=/tmp/mdadm.conf --force
[...]
mdadm: no uptodate device for slot 1 of /dev/md0
mdadm: no uptodate device for slot 2 of /dev/md0
[...]

Regards,

Leon.

Re: Multiple disk failure, but slot numbers are corrupt and preventing assembly.

[Leon Woestenberg]

On 4/24/07, Leon Woestenberg <leon.woestenberg@gmail.com> wrote:
> Hello,
>
> On 4/23/07, David Greaves <david@dgreaves.com> wrote:
> > There is some odd stuff in there:
> >
> [root@localhost ~]# mdadm -v --assemble --scan --config=/tmp/mdadm.conf --force
> [...]
> mdadm: no uptodate device for slot 1 of /dev/md0
> mdadm: no uptodate device for slot 2 of /dev/md0
> [...]
>
So, the problem I am facing is that the slot number (as seen with
--examine) is invalid on two and therefore they won't be recognized as
valid drives for the array.

Is there any way to override the slot number? I could not find
anything in mdadm or mdadm.conf to override them.

-- 
Leon

Re: Multiple disk failure, but slot numbers are corrupt and preventing assembly.

[David Greaves]

Leon Woestenberg wrote:
> On 4/24/07, Leon Woestenberg <leon.woestenberg@gmail.com> wrote:
>> Hello,
>>
>> On 4/23/07, David Greaves <david@dgreaves.com> wrote:
>> > There is some odd stuff in there:
>> >
>> [root@localhost ~]# mdadm -v --assemble --scan
>> --config=/tmp/mdadm.conf --force
>> [...]
>> mdadm: no uptodate device for slot 1 of /dev/md0
>> mdadm: no uptodate device for slot 2 of /dev/md0
>> [...]
>>
> So, the problem I am facing is that the slot number (as seen with
> --examine) is invalid on two and therefore they won't be recognized as
> valid drives for the array.
> 
> Is there any way to override the slot number? I could not find
> anything in mdadm or mdadm.conf to override them.
Yes --create, see my original reply.

Essentially all --create does is create superblocks with the data you want (eg
slot numbers). It does not touch other 'on disk data'.
It is safe to run the *exact same* create command on a dormant array at any time
after initial creation - the main side effect is a new UUID.
(Neil - yell if I'm wrong).

The most 'dangerous' part is to create a superblock with a different version.

If you wanted to experiment (maybe with loopback devices) you could try
--create'ing an array with 4 devices to simulate where you were.
Then do the --create again with 2 devices missing. This should end up with 2
devices with one UUID, 2 with another.

Then do an --assemble using --force and --update=uuid.

Report back if you do this...

Cheers
David

Re: Multiple disk failure, but slot numbers are corrupt and preventing assembly.

[Leon Woestenberg]

David,

thanks for all the advice so far.

On 4/24/07, David Greaves <david@dgreaves.com> wrote:
> Leon Woestenberg wrote:
> > On 4/24/07, Leon Woestenberg <leon.woestenberg@gmail.com> wrote:
> >> Hello,
> >>
> >> On 4/23/07, David Greaves <david@dgreaves.com> wrote:
> >> > There is some odd stuff in there:
> >> >
> >> [root@localhost ~]# mdadm -v --assemble --scan
> >> --config=/tmp/mdadm.conf --force
> >> [...]
> >> mdadm: no uptodate device for slot 1 of /dev/md0
> >> mdadm: no uptodate device for slot 2 of /dev/md0
> >> [...]
> >>
> > So, the problem I am facing is that the slot number (as seen with
> > --examine) is invalid on two and therefore they won't be recognized as
> > valid drives for the array.
> >
> > Is there any way to override the slot number? I could not find
> > anything in mdadm or mdadm.conf to override them.
> Yes --create, see my original reply.
>
> Essentially all --create does is create superblocks with the data you want (eg
> slot numbers). It does not touch other 'on disk data'.
> It is safe to run the *exact same* create command on a dormant array at any time
> after initial creation - the main side effect is a new UUID.
> (Neil - yell if I'm wrong).
>
In first instance we were searching for ways to tell mdadm what we
know about the array (through mdadm.conf) but from all advice we got
we have to take the 'usual' non-syncing-recreate approach.

We will try to make disk clones first. Will dd suffice or do I need
something more fancy that maybe copes with source drive read errors in
a better fashion?

Thanks,
-- 
Leon

Re: Multiple disk failure, but slot numbers are corrupt and preventing assembly.

[David Greaves]

Leon Woestenberg wrote:
> David,
> 
> thanks for all the advice so far.

No problem :)

> In first instance we were searching for ways to tell mdadm what we
> know about the array (through mdadm.conf) but from all advice we got
> we have to take the 'usual' non-syncing-recreate approach.
> 
> We will try to make disk clones first. Will dd suffice or do I need
> something more fancy that maybe copes with source drive read errors in
> a better fashion?

ddrescue and dd_rescue are *much* better.

I favour the gnu ddrescue - it's much easier. But sometimes, on some kernels
with some hardware I've had kernel locks that dd_rescue (eventually, after many
minutes) times out from.

The RIP iso is a good place to start.
http://www.tux.org/pub/people/kent-robotti/looplinux/rip/

David

Re: Multiple disk failure, but slot numbers are corrupt and preventing assembly.

[Bill Davidsen]

Leon Woestenberg wrote:
> In first instance we were searching for ways to tell mdadm what we
> know about the array (through mdadm.conf) but from all advice we got
> we have to take the 'usual' non-syncing-recreate approach.
>
> We will try to make disk clones first. Will dd suffice or do I need
> something more fancy that maybe copes with source drive read errors in
> a better fashion? 

Yes to both. dd will be fine in most cases, and I suggest using noerror 
to continue after errors, and oflag=direct just for performance. You 
could use ddrescue, it supposedly copes better with errors, although I 
don't know details.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: Multiple disk failure, but slot numbers are corrupt and preventing assembly.

[David Greaves]

Bill Davidsen wrote:
> Leon Woestenberg wrote:
>> We will try to make disk clones first. Will dd suffice or do I need
>> something more fancy that maybe copes with source drive read errors in
>> a better fashion? 
> 
> Yes to both. dd will be fine in most cases, and I suggest using noerror
> to continue after errors, and oflag=direct just for performance. You
> could use ddrescue, it supposedly copes better with errors, although I
> don't know details.
> 

Hi Bill

IIRC dd will continue to operate on error but just retries a set number of times
and continues to the next sector.

ddrescue does clever things like jumping a few sectors when it hits an error,
then, after it has retrieved as much data off the disk as possible it starts to
bisect the error area.

This type of algorithm is good because disks often die after a few minutes of
being powered and deteriorate rapidly as data is recovered - hammering them on
retries on an error on sector 312, 313, 314.... when you have millions of
(currently) readable sectors elsewhere is a bad idea because it can minimise the
time you have left to read your data.

It's also very fast, provides continuous updates to the screen as to where it is
and the io rates it is currently seeing and those it's averaging, it also writes
a log of what it's done to local file to allow restarts.
etc etc etcc..
try it - IMHO it's the right tool for the (data-recovery) job :)

David

Re: Multiple disk failure, but slot numbers are corrupt and preventing assembly.

[Leon Woestenberg]

Hello all,

On 4/26/07, David Greaves <david@dgreaves.com> wrote:
> Bill Davidsen wrote:
> > Leon Woestenberg wrote:
> >> We will try to make disk clones first. Will dd suffice or do I need
> >> something more fancy that maybe copes with source drive read errors in
> >> a better fashion?
> >
> > Yes to both. dd will be fine in most cases, and I suggest using noerror
> > to continue after errors, and oflag=direct just for performance. You
> > could use ddrescue, it supposedly copes better with errors, although I
>

OK, despite that we missed a deadline to present the on-disk data by
one day, one of my colleagues flew in to the remote machine,
dd_rescue'd four 500 GB disks to clones, cloned them again at the lab,
then did the 'mdadm create force missing' approach. Everything mounted
read-only cleanly and we could extract the data.

Thanks for helping us out on the details (Specify missing to not sync,
and --force to prevent spares, dd_rescue instead of dd). Especially
helping understand what happens at each step is precious.

Next time we will be more confidently doing this over SSH directly on
the subject, unless data recovery has highest priority and we take the
disk-over-airplane approach.

Kudo's!

Regards, Leon.

Re: Multiple disk failure, but slot numbers are corrupt and preventing assembly.

[Bill Davidsen]

David Greaves wrote:
> Bill Davidsen wrote:
>   
>> Leon Woestenberg wrote:
>>     
>>> We will try to make disk clones first. Will dd suffice or do I need
>>> something more fancy that maybe copes with source drive read errors in
>>> a better fashion? 
>>>       
>> Yes to both. dd will be fine in most cases, and I suggest using noerror
>> to continue after errors, and oflag=direct just for performance. You
>> could use ddrescue, it supposedly copes better with errors, although I
>> don't know details.
>>
>>     
>
> Hi Bill
>
> IIRC dd will continue to operate on error but just retries a set number of times
> and continues to the next sector.
>
>   
As I read the O.P. that's fine in this case, two drives believed just 
fine, one showing okay in SMART, all should copy with dd. The last drive 
is probably not recoverable, and if the other three are recovered RAID5 
will recover in any case without it.
> ddrescue does clever things like jumping a few sectors when it hits an error,
> then, after it has retrieved as much data off the disk as possible it starts to
> bisect the error area.
>
> This type of algorithm is good because disks often die after a few minutes of
> being powered and deteriorate rapidly as data is recovered - hammering them on
> retries on an error on sector 312, 313, 314.... when you have millions of
> (currently) readable sectors elsewhere is a bad idea because it can minimise the
> time you have left to read your data. It's also very fast, provides continuous updates to the screen as to where it is and the io rates it is currently seeing and those it's averaging, it also writes
> a log of what it's done to local file to allow restarts.
> etc etc etcc..
> try it - IMHO it's the right tool for the (data-recovery) job :)
>   
I won't recommend a tool I haven't tried, and in this case I don't think 
is going to be needed. I mentioned it existed, that's as far as I want 
to go. I wouldn't suggest someone else use a tool I haven't used (or 
written) myself.

If you've used it and found it worked on a drive with actual bad spots, 
you offered your information which is appropriate.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: Multiple disk failure, but slot numbers are corrupt and preventing assembly.

[Neil Brown]

On Tuesday April 24, leon.woestenberg@gmail.com wrote:
> David,
> 
> thanks for all the advice so far.
> 
> On 4/24/07, David Greaves <david@dgreaves.com> wrote:
> >
> > Essentially all --create does is create superblocks with the data you want (eg
> > slot numbers). It does not touch other 'on disk data'.
> > It is safe to run the *exact same* create command on a dormant array at any time
> > after initial creation - the main side effect is a new UUID.
> > (Neil - yell if I'm wrong).

Close enough.  The "active/clean" flag also gets changed.
And for raid5, you want "--force" or it will make a degraded array
with one spare which may not be what you want.


> >
> In first instance we were searching for ways to tell mdadm what we
> know about the array (through mdadm.conf) but from all advice we got

"mdadm.conf" doesn't really know anything about the shape of the
array.  It just describes how to discover and recognise component
devices.   The metadata on the device is authoritative, and if it is
corrupt.... it is corrupt.

> we have to take the 'usual' non-syncing-recreate approach.

If "mdadm -f" doesn't work, then there is rarely any other option but
re-creating the array.  If you a reasonably careful the data will
still be there (unless it was corrupted by whatever corrupted the
metadata). 

NeilBrown