RAID5 disk failure during rebuild of spare, any chance of recovery when one of the failed devices is suspected to be intact?

RAID5 disk failure during rebuild of spare, any chance of recovery when one of the failed devices is suspected to be intact?

[Tor Arne Vestbø]

Hey,

Hoping for some crisis help here :)

Array consists of /dev/sd[bcdef]1 where b-e were active devices and
sdf1 was a spare.

After installing Ubunuty 10.04 and trying to reassemble the array it
got reassembled without sdb1, so mdadm started reconstructing the
array onto the spare sdf1. While this was going on, sdd failed, and
was pulled out as faulty. Now things look like this:

# mdadm --detail /dev/md0
/dev/md0:
        Version : 00.90
  Creation Time : Sun Mar  2 22:52:53 2008
     Raid Level : raid5
     Array Size : 2197715712 (2095.91 GiB 2250.46 GB)
  Used Dev Size : 732571904 (698.64 GiB 750.15 GB)
   Raid Devices : 4
  Total Devices : 5
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Sun Aug 15 20:32:59 2010
          State : clean, degraded
 Active Devices : 2
Working Devices : 4
 Failed Devices : 1
  Spare Devices : 2

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : a0186556:4ffb5a2a:822f8875:94ae7d2c
         Events : 0.24708

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8       33        1      active sync   /dev/sdc1
       2       0        0        2      removed
       3       8       65        3      active sync   /dev/sde1

       4       8       17        -      spare   /dev/sdb1
       5       8       81        -      spare   /dev/sdf1
       6       8       49        -      faulty spare   /dev/sdd1


Mounting the array does not work :/

Normally a RAID5 with two lost devices is unrecoverable, as far as
I've understood it, but in this case i suspect that sdb1 is fully
intact, and that it was just for some reason picked up when the array
was first assembled.

If that's the case, is there any way i can "promote" sdb1 from spare
to active without rebuilding it (which would not work since the array
is messed up)? Basically reassembling the array as if sdb1, sdc1 and
sde1 were okey? and then rebuild the sdf1 spare?

Thanks!

Re: RAID5 disk failure during rebuild of spare, any chance of recovery when one of the failed devices is suspected to be intact?

[Tor Arne Vestbø]

On Sun, Aug 15, 2010 at 8:52 PM, Tor Arne Vestbø <torarnv@gmail.com> wrote:
> Hey,
>
> Hoping for some crisis help here :)

Here's a few logs if it helps...

/var/log/messages --> http://pastebin.com/mwny786t

You can see sdb1 being kicked out at some point due to being "non-fresh"

dmesg --> http://pastebin.com/073ws32r

/var/log/syslog  --> http://pastebin.com/GarHJD4c

mdadm -E /dev/sd[bcdef]1 --> http://pastebin.com/Kp145Mkx

Thanks!
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID5 disk failure during rebuild of spare, any chance of recovery when one of the failed devices is suspected to be intact?

[Tor Arne Vestbø]

On Sun, Aug 15, 2010 at 10:06 PM, Tor Arne Vestbø <torarnv@gmail.com> wrote:
> On Sun, Aug 15, 2010 at 8:52 PM, Tor Arne Vestbø <torarnv@gmail.com> wrote:
>> Hey,
>>
>> Hoping for some crisis help here :)
>
> Here's a few logs if it helps...

After reading various tips on recovering I tried the following:

# mdadm -Afv /dev/md0 /dev/sdb1 /dev/sdc1 /dev/sde1
mdadm: looking for devices for /dev/md0
mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 4.
mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 3.
mdadm: no uptodate device for slot 0 of /dev/md0
mdadm: no uptodate device for slot 2 of /dev/md0
mdadm: added /dev/sde1 to /dev/md0 as 3
mdadm: added /dev/sdb1 to /dev/md0 as 4
mdadm: added /dev/sdc1 to /dev/md0 as 1
mdadm: /dev/md0 assembled from 2 drives and 1 spare - not enough to
start the array.

Thinking that since sdb1 was more or less intact it would be able to
jump in and work together with sdc1 and sde1 to form a degraded array
I could pull data off of.

But it seems that when I added sdb1 to the array earlier after it
initially was kicked for being "non-fresh", it got added as a spare,
in slot 4.

The message about no uptodoate device for slot 2 is correct, that's
the sdd1 device that failed. But for slot 0 it should be sdb1.

Is ther any way I can tell mdadm to assume sdb1 is up to date, and use
that at slot 0?

Thanks!

Tor Arne

PS: Pastebin.com seems to be down, so here's a repaste of mdadm -E:
http://pastebin.org/518816
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID5 disk failure during rebuild of spare, any chance of recovery when one of the failed devices is suspected to be intact?

[Nicolas Jungers]

I'd try to recreate the array with a copy of the disks

do a ddrescue of the 3 good disks and then try a mdadm -C with the exact 
parameters of the array creation with the same mdadm version (use 
missing for slot 2).  It saved my raid10 array.

N.


On 08/16/2010 12:33 AM, Tor Arne Vestbø wrote:
> On Sun, Aug 15, 2010 at 10:06 PM, Tor Arne Vestbø<torarnv@gmail.com>  wrote:
>> On Sun, Aug 15, 2010 at 8:52 PM, Tor Arne Vestbø<torarnv@gmail.com>  wrote:
>>> Hey,
>>>
>>> Hoping for some crisis help here :)
>>
>> Here's a few logs if it helps...
>
> After reading various tips on recovering I tried the following:
>
> # mdadm -Afv /dev/md0 /dev/sdb1 /dev/sdc1 /dev/sde1
> mdadm: looking for devices for /dev/md0
> mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 4.
> mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot 1.
> mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 3.
> mdadm: no uptodate device for slot 0 of /dev/md0
> mdadm: no uptodate device for slot 2 of /dev/md0
> mdadm: added /dev/sde1 to /dev/md0 as 3
> mdadm: added /dev/sdb1 to /dev/md0 as 4
> mdadm: added /dev/sdc1 to /dev/md0 as 1
> mdadm: /dev/md0 assembled from 2 drives and 1 spare - not enough to
> start the array.
>
> Thinking that since sdb1 was more or less intact it would be able to
> jump in and work together with sdc1 and sde1 to form a degraded array
> I could pull data off of.
>
> But it seems that when I added sdb1 to the array earlier after it
> initially was kicked for being "non-fresh", it got added as a spare,
> in slot 4.
>
> The message about no uptodoate device for slot 2 is correct, that's
> the sdd1 device that failed. But for slot 0 it should be sdb1.
>
> Is ther any way I can tell mdadm to assume sdb1 is up to date, and use
> that at slot 0?
>
> Thanks!
>
> Tor Arne
>
> PS: Pastebin.com seems to be down, so here's a repaste of mdadm -E:
> http://pastebin.org/518816
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID5 disk failure during rebuild of spare, any chance of recovery when one of the failed devices is suspected to be intact?

[Tor Arne Vestbø]

On Mon, Aug 16, 2010 at 12:33 AM, Tor Arne Vestbø <torarnv@gmail.com> wrote:
> Thinking that since sdb1 was more or less intact it would be able to
> jump in and work together with sdc1 and sde1 to form a degraded array
> I could pull data off of.

Here's a paste of the mdadm -E state of things before this happened:

http://pastebin.com/f46EWXEA

I notice the event count has gone up since then, does this indicate
that sdb1 will be too old (non-fresh) to be usable?

Perhaps I'll have more luck assembling the array with the known-faulty
sdd1 with a little bit of file corruption, or perhaps it was just
failing temporarily and got kicked just to be safe?

Tor Arne
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID5 disk failure during rebuild of spare, any chance of recovery when one of the failed devices is suspected to be intact?

[Tor Arne Vestbø]

Hey Nicolas!

On Mon, Aug 16, 2010 at 7:29 AM, Nicolas Jungers <nicolas@jungers.net> wrote:
> I'd try to recreate the array with a copy of the disks
>
> do a ddrescue of the 3 good disks and then try a mdadm -C with the exact
> parameters of the array creation with the same mdadm version (use missing
> for slot 2).  It saved my raid10 array.

You mean you sdc and sde plus either sdb or sdd, depending on which
one I think is more sane a this point?

mdadm -E before http://pastebin.com/f46EWXEA vs after
http://pastebin.com/Kp145Mkx indicates sdb1 is missing a few events,
dunno if that's too much to be usable at this point?

Can the create-process be done "read-only", so that I don't have to
use extra disks? If I've understood things correctly --create will
only overwrite the superblocks, not touch the data?

So I could do:

mdadm -C /dev/md0 /dev/sdb1 /dev/sdc1 misssing /dev/sde1

I suppose the 33% restored spare sdf1 won't help any in this situation?

Tor Arne
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID5 disk failure during rebuild of spare, any chance of recovery when one of the failed devices is suspected to be intact?

[Nicolas Jungers]

On 08/16/2010 07:54 AM, Tor Arne Vestbø wrote:
> Hey Nicolas!
>
> On Mon, Aug 16, 2010 at 7:29 AM, Nicolas Jungers<nicolas@jungers.net>  wrote:
>> I'd try to recreate the array with a copy of the disks
>>
>> do a ddrescue of the 3 good disks and then try a mdadm -C with the exact
>> parameters of the array creation with the same mdadm version (use missing
>> for slot 2).  It saved my raid10 array.
>
> You mean you sdc and sde plus either sdb or sdd, depending on which
> one I think is more sane a this point?

I'd try both.  Do a ddrescue of the failing one and try that (with copy 
of the others) and check what's coming out.

>
> mdadm -E before http://pastebin.com/f46EWXEA vs after
> http://pastebin.com/Kp145Mkx indicates sdb1 is missing a few events,
> dunno if that's too much to be usable at this point?

Did write happen during the reconstruction?  You'll lose not only what 
was on the write but also everything on the same strip, including 
directories.  That's a reason to try with a copy of sdd.

> Can the create-process be done "read-only", so that I don't have to
> use extra disks? If I've understood things correctly --create will
> only overwrite the superblocks, not touch the data?

yes, but I'd not do it.  And beside, you should not trust the failing 
one (sdd).

>
> So I could do:
>
> mdadm -C /dev/md0 /dev/sdb1 /dev/sdc1 misssing /dev/sde1

or even
mdadm -C /dev/md0 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1

with copies.

>
> I suppose the 33% restored spare sdf1 won't help any in this situation?

I suppose not.

N.
>
> Tor Arne

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID5 disk failure during rebuild of spare, any chance of recovery when one of the failed devices is suspected to be intact?

[Tim Small]

On 16/08/10 07:12, Nicolas Jungers wrote:
> On 08/16/2010 07:54 AM, Tor Arne Vestbø wrote:
>> You mean you sdc and sde plus either sdb or sdd, depending on which
>> one I think is more sane a this point?
>
> I'd try both.  Do a ddrescue of the failing one and try that (with 
> copy of the others) and check what's coming out.

As an alternative to using ddrescue, you could quickly prototype various 
arrangements (without writing anything to the drives) using a 
device-mapper copy-on-write mapping - I posted some details to the list 
a while back when I was trying to use this to reconstruct a hw raid 
array...  Check the list archives for details.

Tim.

-- 
South East Open Source Solutions Limited
Registered in England and Wales with company number 06134732.
Registered Office: 2 Powell Gardens, Redhill, Surrey, RH1 1TQ
VAT number: 900 6633 53  http://seoss.co.uk/ +44-(0)1273-808309

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID5 disk failure during rebuild of spare, any chance of recovery when one of the failed devices is suspected to be intact?

[Tor Arne Vestbø]

On 16.08.10 08.12, Nicolas Jungers wrote:
>> mdadm -E before http://pastebin.com/f46EWXEA vs after
>> http://pastebin.com/Kp145Mkx indicates sdb1 is missing a few events,
>> dunno if that's too much to be usable at this point?
>
> Did write happen during the reconstruction? You'll lose not only what
> was on the write but also everything on the same strip, including
> directories. That's a reason to try with a copy of sdd.

I've got a LVM volume group on the array, but never mounted any of the 
logical volumes during the reconstruction. Not sure if LVM will write 
something in the background that would have messed up things?

Tor Arne

Re: RAID5 disk failure during rebuild of spare, any chance of recovery when one of the failed devices is suspected to be intact?

[Tor Arne Vestbø]

On Mon, Aug 16, 2010 at 10:43 AM, Tim Small <tim@seoss.co.uk> wrote:
> On 16/08/10 07:12, Nicolas Jungers wrote:
>>
>> On 08/16/2010 07:54 AM, Tor Arne Vestbø wrote:
>>>
>>> You mean you sdc and sde plus either sdb or sdd, depending on which
>>> one I think is more sane a this point?
>>
>> I'd try both.  Do a ddrescue of the failing one and try that (with copy of
>> the others) and check what's coming out.
>
> As an alternative to using ddrescue, you could quickly prototype various
> arrangements (without writing anything to the drives) using a device-mapper
> copy-on-write mapping - I posted some details to the list a while back when
> I was trying to use this to reconstruct a hw raid array...  Check the list
> archives for details.

Cool, here's what I tried:

Created spares files for each of the devices

  dd if=/dev/zero of=sdb_cow bs=1 count=0 seek=2GB

Mapped that to a loop device

  losetup /dev/loop1 sdb_cow

Then ran the following for each device:

  cow_size=`blockdev --getsize /dev/sdb1`
  chunk_size=64
  echo "0 $cow_size snapshot /dev/sdb1 /dev/loop1 p $chunk_size" |
dmsetup create sdb1_cow

After these were created I tried the following:

# mdadm -v -C /dev/md0 -l5 -n4 /dev/mapper/sdb1_cow
/dev/mapper/sdc1_cow missing /dev/mapper/sde1_cow
mdadm: layout defaults to left-symmetric
mdadm: chunk size defaults to 64K
mdadm: /dev/mapper/sdb1_cow appears to be part of a raid array:
    level=raid5 devices=4 ctime=Sun Mar  2 22:52:53 2008
mdadm: /dev/mapper/sdc1_cow appears to be part of a raid array:
    level=raid5 devices=4 ctime=Sun Mar  2 22:52:53 2008
mdadm: /dev/mapper/sde1_cow appears to be part of a raid array:
    level=raid5 devices=4 ctime=Sun Mar  2 22:52:53 2008
mdadm: size set to 732571904K
Continue creating array? Y
mdadm: array /dev/md0 started.

# mdadm --detail /dev/md0
/dev/md0:
        Version : 00.90
  Creation Time : Mon Aug 16 18:20:06 2010
     Raid Level : raid5
     Array Size : 2197715712 (2095.91 GiB 2250.46 GB)
  Used Dev Size : 732571904 (698.64 GiB 750.15 GB)
   Raid Devices : 4
  Total Devices : 3
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Mon Aug 16 18:20:06 2010
          State : clean, degraded
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : 916ceaa2:b877a3cc:3973abef:31f2d600 (local to host monstre)
         Events : 0.1

    Number   Major   Minor   RaidDevice State
       0     251        9        0      active sync   /dev/block/251:9
       1     251       10        1      active sync   /dev/block/251:10
       2       0        0        2      removed
       3     251       12        3      active sync   /dev/block/251:12

And I can now mount /dev/mapper/raid-home !

The question now is, what next? Should I start copying things off to a
backup, or run fsck first or something else to try to repair errors?
Or perhaps are the 2GB sparse files to small for anything like that?

Thanks!

Tor Arne
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID5 disk failure during rebuild of spare, any chance of recovery when one of the failed devices is suspected to be intact?

[Nicolas Jungers]

On 08/16/2010 06:27 PM, Tor Arne Vestbø wrote:
> On Mon, Aug 16, 2010 at 10:43 AM, Tim Small<tim@seoss.co.uk>  wrote:
>> On 16/08/10 07:12, Nicolas Jungers wrote:
>>>
>>> On 08/16/2010 07:54 AM, Tor Arne Vestbø wrote:
>>>>
>>>> You mean you sdc and sde plus either sdb or sdd, depending on which
>>>> one I think is more sane a this point?
>>>
>>> I'd try both.  Do a ddrescue of the failing one and try that (with copy of
>>> the others) and check what's coming out.
>>
>> As an alternative to using ddrescue, you could quickly prototype various
>> arrangements (without writing anything to the drives) using a device-mapper
>> copy-on-write mapping - I posted some details to the list a while back when
>> I was trying to use this to reconstruct a hw raid array...  Check the list
>> archives for details.
>
> Cool, here's what I tried:
>
> Created spares files for each of the devices
>
>    dd if=/dev/zero of=sdb_cow bs=1 count=0 seek=2GB
>
> Mapped that to a loop device
>
>    losetup /dev/loop1 sdb_cow
>
> Then ran the following for each device:
>
>    cow_size=`blockdev --getsize /dev/sdb1`
>    chunk_size=64
>    echo "0 $cow_size snapshot /dev/sdb1 /dev/loop1 p $chunk_size" |
> dmsetup create sdb1_cow
>
> After these were created I tried the following:
>
> # mdadm -v -C /dev/md0 -l5 -n4 /dev/mapper/sdb1_cow
> /dev/mapper/sdc1_cow missing /dev/mapper/sde1_cow
> mdadm: layout defaults to left-symmetric
> mdadm: chunk size defaults to 64K
> mdadm: /dev/mapper/sdb1_cow appears to be part of a raid array:
>      level=raid5 devices=4 ctime=Sun Mar  2 22:52:53 2008
> mdadm: /dev/mapper/sdc1_cow appears to be part of a raid array:
>      level=raid5 devices=4 ctime=Sun Mar  2 22:52:53 2008
> mdadm: /dev/mapper/sde1_cow appears to be part of a raid array:
>      level=raid5 devices=4 ctime=Sun Mar  2 22:52:53 2008
> mdadm: size set to 732571904K
> Continue creating array? Y
> mdadm: array /dev/md0 started.
>
> # mdadm --detail /dev/md0
> /dev/md0:
>          Version : 00.90
>    Creation Time : Mon Aug 16 18:20:06 2010
>       Raid Level : raid5
>       Array Size : 2197715712 (2095.91 GiB 2250.46 GB)
>    Used Dev Size : 732571904 (698.64 GiB 750.15 GB)
>     Raid Devices : 4
>    Total Devices : 3
> Preferred Minor : 0
>      Persistence : Superblock is persistent
>
>      Update Time : Mon Aug 16 18:20:06 2010
>            State : clean, degraded
>   Active Devices : 3
> Working Devices : 3
>   Failed Devices : 0
>    Spare Devices : 0
>
>           Layout : left-symmetric
>       Chunk Size : 64K
>
>             UUID : 916ceaa2:b877a3cc:3973abef:31f2d600 (local to host monstre)
>           Events : 0.1
>
>      Number   Major   Minor   RaidDevice State
>         0     251        9        0      active sync   /dev/block/251:9
>         1     251       10        1      active sync   /dev/block/251:10
>         2       0        0        2      removed
>         3     251       12        3      active sync   /dev/block/251:12
>
> And I can now mount /dev/mapper/raid-home !
>
> The question now is, what next? Should I start copying things off to a
> backup, or run fsck first or something else to try to repair errors?
> Or perhaps are the 2GB sparse files to small for anything like that?

For me: first, copy everything.  You have an unreliable disk in the 
middle of your data.

N.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html