help recovering a software raid5 device

help recovering a software raid5 device

[Theo Cabrerizo Diem]

Hello,

I had (hopefully still have ;) ) a raid5 setup using 4x 2Tb HDDs.
Recently when I was on vacations one disk failed and before I could
come back (15 days after) a second one failed also.

Of course I had no backups ;) and to make it worse the raid device had
luks running on top of it, but that would come later. Just as
clarfication I did encrypt the md device, and not built a md on top of
luks devices.

I did read the wiki, and took a copy of mdadm --examine /dev/sd[ghij]1
before doing anything. I've tried to run :

mdadm --create --assume-clean --level=5 --chunk 64 --raid-devices=4
/dev/md/stuff1 /dev/sdh1 /dev/sdg1 /dev/sdj1 /dev/sdi1

Including the "supposed" correct order taken from examine command and
all possible permutations of the disk orders with no luck.

From there I have three questions :
- From the output of --examine I have "Chunk Size: 64K". I suppose I
should add "--chunk 64" on the mdadm --create line right?
- Should I attempt "mdadm --create" command with just the last 3 good
disks and a "missing" one or should I attempt with all four ?
- Any further suggestions to try to recover it ?

Following my output of mdadm --examine after a reboot (don't know why
the distro detected and assembled the raid with only two devices in a
inactive state)

Thanks in advance,


=== mdadm --examine ===
/dev/sdg1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 9e0221c0:4c86d48e:17f44c71:374ab4fc
           Name : turquoise:stuff1  (local to host turquoise)
  Creation Time : Thu Apr  7 21:53:47 2011
     Raid Level : raid5
   Raid Devices : 4
 Avail Dev Size : 3907024897 (1863.01 GiB 2000.40 GB)
     Array Size : 5860536768 (5589.04 GiB 6001.19 GB)
  Used Dev Size : 3907024512 (1863.01 GiB 2000.40 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : b126fff4:3072b6ad:70f6d710:9211b38a
    Update Time : Sun Jan  6 13:47:04 2013
       Checksum : 9ea2be38 - correct
         Events : 196828
         Layout : left-symmetric
     Chunk Size : 64K
   Device Role : Active device 1
   Array State : AAAA ('A' == active, '.' == missing)
/dev/sdh1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 9e0221c0:4c86d48e:17f44c71:374ab4fc
           Name : turquoise:stuff1  (local to host turquoise)
  Creation Time : Thu Apr  7 21:53:47 2011
     Raid Level : raid5
   Raid Devices : 4
 Avail Dev Size : 3907024897 (1863.01 GiB 2000.40 GB)
     Array Size : 5860536768 (5589.04 GiB 6001.19 GB)
  Used Dev Size : 3907024512 (1863.01 GiB 2000.40 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 7429aae9:e6b533db:b8691ef5:82776fe3
    Update Time : Sun Jan 20 06:26:40 2013
       Checksum : fde2d905 - correct
         Events : 323123
         Layout : left-symmetric
     Chunk Size : 64K
   Device Role : Active device 0
   Array State : A.AA ('A' == active, '.' == missing)
/dev/sdi1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 9e0221c0:4c86d48e:17f44c71:374ab4fc
           Name : turquoise:stuff1  (local to host turquoise)
  Creation Time : Thu Apr  7 21:53:47 2011
     Raid Level : raid5
   Raid Devices : 4
 Avail Dev Size : 3907024897 (1863.01 GiB 2000.40 GB)
     Array Size : 5860536768 (5589.04 GiB 6001.19 GB)
  Used Dev Size : 3907024512 (1863.01 GiB 2000.40 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 0cd288e4:91fe05b3:2635c653:47d1b20c
    Update Time : Sun Jan 27 10:10:06 2013
       Checksum : 58865ead - correct
         Events : 323208
         Layout : left-symmetric
     Chunk Size : 64K
   Device Role : Active device 3
   Array State : ..AA ('A' == active, '.' == missing)
/dev/sdj1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 9e0221c0:4c86d48e:17f44c71:374ab4fc
           Name : turquoise:stuff1  (local to host turquoise)
  Creation Time : Thu Apr  7 21:53:47 2011
     Raid Level : raid5
   Raid Devices : 4
 Avail Dev Size : 3907024897 (1863.01 GiB 2000.40 GB)
     Array Size : 5860536768 (5589.04 GiB 6001.19 GB)
  Used Dev Size : 3907024512 (1863.01 GiB 2000.40 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 69c3570d:57b1ef84:7eb3bac5:ba27f173
    Update Time : Sun Jan 27 10:10:06 2013
       Checksum : 2c71d799 - correct
         Events : 323208
         Layout : left-symmetric
     Chunk Size : 64K
   Device Role : Active device 2
   Array State : ..AA ('A' == active, '.' == missing)
=== end ===

Re: help recovering a software raid5 device

[Phil Turmel]

On 01/27/2013 08:52 AM, Theo Cabrerizo Diem wrote:
> Hello,
> 
> I had (hopefully still have ;) ) a raid5 setup using 4x 2Tb HDDs.
> Recently when I was on vacations one disk failed and before I could
> come back (15 days after) a second one failed also.
> 
> Of course I had no backups ;) and to make it worse the raid device had
> luks running on top of it, but that would come later. Just as
> clarfication I did encrypt the md device, and not built a md on top of
> luks devices.
> 
> I did read the wiki, and took a copy of mdadm --examine /dev/sd[ghij]1
> before doing anything. I've tried to run :
> 
> mdadm --create --assume-clean --level=5 --chunk 64 --raid-devices=4
> /dev/md/stuff1 /dev/sdh1 /dev/sdg1 /dev/sdj1 /dev/sdi1

For some reason, people are unwilling to use "--assemble --force", which
is made for these situations.

This is the correct device order, though, so you aren't toast yet.

> Including the "supposed" correct order taken from examine command and
> all possible permutations of the disk orders with no luck.
> 
> From there I have three questions :
> - From the output of --examine I have "Chunk Size: 64K". I suppose I
> should add "--chunk 64" on the mdadm --create line right?

Yes.

> - Should I attempt "mdadm --create" command with just the last 3 good
> disks and a "missing" one or should I attempt with all four ?
> - Any further suggestions to try to recover it ?

I would leave out the disk that failed first (/dev/sdg1, I believe).
Presumably there was still some activity on the system?

> Following my output of mdadm --examine after a reboot (don't know why
> the distro detected and assembled the raid with only two devices in a
> inactive state)

The appended --examine reports show a creation time from 2011, but an
update time from just a little while ago.  Did you cancel the "--create"
operation(s)?  (That would be good, actually.)

Please show the saved "--examine" reports, and current "--examine" reports.

It wouldn't hurt to also post the "smartctl -x" for each of these drives.

Phil

Re: help recovering a software raid5 device

[Keith Keller]

On 2013-01-28, Phil Turmel <philip@turmel.org> wrote:
> On 01/27/2013 08:52 AM, Theo Cabrerizo Diem wrote:
>> 
>> I did read the wiki, and took a copy of mdadm --examine /dev/sd[ghij]1
>> before doing anything. I've tried to run :
>> 
>> mdadm --create --assume-clean --level=5 --chunk 64 --raid-devices=4
>> /dev/md/stuff1 /dev/sdh1 /dev/sdg1 /dev/sdj1 /dev/sdi1
>
> For some reason, people are unwilling to use "--assemble --force", which
> is made for these situations.

It is because the wiki suggests the --create --assume-clean strategy:

https://raid.wiki.kernel.org/index.php/RAID_Recovery

I thought I modified that page, but perhaps I am misremembering, or
perhaps I wasn't sure of the best advice to put there.

--keith

-- 
kkeller@wombat.san-francisco.ca.us

Re: help recovering a software raid5 device

[Phil Turmel]

Hi Theo,

[list restored--please use reply-to-all for kernel.org lists]

On 01/28/2013 05:04 AM, Theo Cabrerizo Diem wrote:
> On 28 January 2013 02:28, Phil Turmel <philip@turmel.org> wrote:
>> On 01/27/2013 08:52 AM, Theo Cabrerizo Diem wrote:
>>> Hello,
> snip
>>>
>>> I did read the wiki, and took a copy of mdadm --examine /dev/sd[ghij]1
>>> before doing anything. I've tried to run :
>>>
>>> mdadm --create --assume-clean --level=5 --chunk 64 --raid-devices=4
>>> /dev/md/stuff1 /dev/sdh1 /dev/sdg1 /dev/sdj1 /dev/sdi1
>>
>> For some reason, people are unwilling to use "--assemble --force", which
>> is made for these situations.
>>
>> This is the correct device order, though, so you aren't toast yet.
>>
> As mentioned by Keith Keller, it is how is instructed on wiki. I had
> the feeling it was not "right" since if you don't add --assume-clean
> it would rebuild it empty, which is fairly dangerous imho ;)
> 
> So before I mess it up even more, the proper command (in my case) would be :
> 
> mdadm --assemble /dev/md/stuff1 --force /dev/sdh1 missing /dev/sdj1 /dev/sdi1
> 
> right ? But I believe the superblock was already overwritten by the
> suggested --create --assume-clean. Should it still be "safe" to try ?

Yes, it is now too late for "--assemble --force".

> I found curious that there's no option to force md to not write
> anything to disks at all, a read-only mechanism for attempting to
> recovery. Any attempt you make potential updates at minimal timestamps
> that could change the original data.

Which is why saving the "--examine" output is so important.

>>> - Should I attempt "mdadm --create" command with just the last 3 good
>>> disks and a "missing" one or should I attempt with all four ?
>>> - Any further suggestions to try to recover it ?
>>
>> I would leave out the disk that failed first (/dev/sdg1, I believe).
>> Presumably there was still some activity on the system?
> 
> Yes, the system was still up but "frozen" since any attempt to access
> the raid device resulted in endless amounts of io error. I've
> attempted an emergency sync and hard booted.

I meant activity between the first failure and the second.

>>> Following my output of mdadm --examine after a reboot (don't know why
>>> the distro detected and assembled the raid with only two devices in a
>>> inactive state)
>>
>> The appended --examine reports show a creation time from 2011, but an
>> update time from just a little while ago.  Did you cancel the "--create"
>> operation(s)?  (That would be good, actually.)
> 
> The examine report was before any attempt of recovery. Unfortunately I
> did run the --create --assume-clean commands as suggested on wiki :(
> ..
> 
>>
>> Please show the saved "--examine" reports, and current "--examine" reports.
> 
> Recent examine report:
> http://pastie.org/5895552
> 
> Saved examine report (same as previously attached):
> http://pastie.org/5895849

In the future, paste these directly into the mail.  Who knows how long
pastie.org will hold on to these, and these mails will be archived
basically forever.

Anyways, they show your problem.

The original reports all have:

>    Data Offset : 2048 sectors

Your recreated array devices have:

>    Data Offset : 262144 sectors

So your copy of mdadm is very new, and has the new defaults for data
offset (leaving more room for a bad block log).  You need to boot with a
slightly older liveCD or other rescue media to get a copy of mdadm that
is about 1 year old.  Re-run the "mdadm --create --assume-clean" with
that version of mdadm.

(The development version of mdadm has command-line syntax to set the
data offset per device, but I don't believe it has been released yet.
If you are comfortable using git and compiling your own utility, that
would be another option.)

>> It wouldn't hurt to also post the "smartctl -x" for each of these drives.
>>
> http://pastie.org/5895385 (sdg - the really broken one - will RMA this
> one after recovering or giving up)

It doesn't appear to be broken.  Just some pending sectors that'll
probably be cleaned up by a wipe, and would have been taken care of with
regular scrubbing.

> http://pastie.org/5895387 (sdh - apparently clean)
> http://pastie.org/5895388 (sdi - apparently clean)
> http://pastie.org/5895389 (sdj - apparently clean)

These do show one critical piece of information that is probably the
only real problem in your system:

> Warning: device does not support SCT Error Recovery Control command

You are using cheap desktop drives that do not support time limits on
error recovery.  They are completely *unsafe* to use "out-of-the-box" in
*any* raid array.

If they did support SCTERC, you could use a boot script to set short
timeouts.  Since they don't, your only option is a boot script to set
very long timeouts in the linux driver for each disk.

> #! /bin/bash
> # Place in rc.local or wherever your distro expects boot-time scripts
> #
> for x in sdg sdh sdi sdj
> do
>     echo 180 >/sys/block/$x/device/timeout
> done

Long timeouts can have negative consequences for services that might be
using the array, but you have no choice.  If you don't do this, any
unrecoverable read error will cause the offending disk to be kicked out
instead of fixed.  (Including errors found during scrubbing.)

> Thanks for stepping up for help :). I did use pastie.org to avoid a
> wall of text. some of those outputs are even bigger what is allowed by
> pastie. Let me know if you would prefer next outputs to be inline.

Yes.

HTH,

Phil

Re: help recovering a software raid5 device

[Theo Cabrerizo Diem]

On 28 January 2013 14:45, Phil Turmel <philip@turmel.org> wrote:
> Hi Theo,
>
> [list restored--please use reply-to-all for kernel.org lists]
>
sorry about that.

> On 01/28/2013 05:04 AM, Theo Cabrerizo Diem wrote:
>> On 28 January 2013 02:28, Phil Turmel <philip@turmel.org> wrote:
>>> On 01/27/2013 08:52 AM, Theo Cabrerizo Diem wrote:
>>>> Hello,
>> snip
>>>>
>>>> I did read the wiki, and took a copy of mdadm --examine /dev/sd[ghij]1
>>>> before doing anything. I've tried to run :
>>>>
>>>> mdadm --create --assume-clean --level=5 --chunk 64 --raid-devices=4
>>>> /dev/md/stuff1 /dev/sdh1 /dev/sdg1 /dev/sdj1 /dev/sdi1
>>>
>>> For some reason, people are unwilling to use "--assemble --force", which
>>> is made for these situations.
>>>
>>> This is the correct device order, though, so you aren't toast yet.
>>>
>> As mentioned by Keith Keller, it is how is instructed on wiki. I had
>> the feeling it was not "right" since if you don't add --assume-clean
>> it would rebuild it empty, which is fairly dangerous imho ;)
>>
>> So before I mess it up even more, the proper command (in my case) would be :
>>
>> mdadm --assemble /dev/md/stuff1 --force /dev/sdh1 missing /dev/sdj1 /dev/sdi1
>>
>> right ? But I believe the superblock was already overwritten by the
>> suggested --create --assume-clean. Should it still be "safe" to try ?
>
> Yes, it is now too late for "--assemble --force".
>
Is there a way that I could flag the raid device (or the partitions)
to not be auto-detected on boot ? I'm afraid that since the "mdadm
--create --assume-clean" completed successfuly before, a reboot on
this machine might bring the array fully online and, for example,
might trigger a check or resync of data. That would be the worse case.

>> I found curious that there's no option to force md to not write
>> anything to disks at all, a read-only mechanism for attempting to
>> recovery. Any attempt you make potential updates at minimal timestamps
>> that could change the original data.
>
> Which is why saving the "--examine" output is so important.
>
>>>> - Should I attempt "mdadm --create" command with just the last 3 good
>>>> disks and a "missing" one or should I attempt with all four ?
>>>> - Any further suggestions to try to recover it ?
>>>
>>> I would leave out the disk that failed first (/dev/sdg1, I believe).
>>> Presumably there was still some activity on the system?
>>
>> Yes, the system was still up but "frozen" since any attempt to access
>> the raid device resulted in endless amounts of io error. I've
>> attempted an emergency sync and hard booted.
>
> I meant activity between the first failure and the second.

Yes, the system was active between the failures, which I've figured
out the mdadm cron mails were bouncing thus the first failure was
unnoticed from my side. Being a sysadmin at work means not always you
have the will to fix everything at home too ;) . Lesson learned.

>
>>>> Following my output of mdadm --examine after a reboot (don't know why
>>>> the distro detected and assembled the raid with only two devices in a
>>>> inactive state)
>>>
>>> The appended --examine reports show a creation time from 2011, but an
>>> update time from just a little while ago.  Did you cancel the "--create"
>>> operation(s)?  (That would be good, actually.)
>>
>> The examine report was before any attempt of recovery. Unfortunately I
>> did run the --create --assume-clean commands as suggested on wiki :(
>> ..
>>
>>>
>>> Please show the saved "--examine" reports, and current "--examine" reports.
>>
>> Recent examine report:
>> http://pastie.org/5895552
>>
>> Saved examine report (same as previously attached):
>> http://pastie.org/5895849
>
> In the future, paste these directly into the mail.  Who knows how long
> pastie.org will hold on to these, and these mails will be archived
> basically forever.
>
> Anyways, they show your problem.
>
> The original reports all have:
>
>>    Data Offset : 2048 sectors
>
> Your recreated array devices have:
>
>>    Data Offset : 262144 sectors

I'm grad to see there is still hope.

>
> So your copy of mdadm is very new, and has the new defaults for data
> offset (leaving more room for a bad block log).  You need to boot with a
> slightly older liveCD or other rescue media to get a copy of mdadm that
> is about 1 year old.  Re-run the "mdadm --create --assume-clean" with
> that version of mdadm.
>
> (The development version of mdadm has command-line syntax to set the
> data offset per device, but I don't believe it has been released yet.
> If you are comfortable using git and compiling your own utility, that
> would be another option.)
>
I have no problem compiling the tools myself. I would actually prefer
that than triggering a reboot on the machine and having unpredictable
results from how it would be detected after the multiple attempts to
create the device.

Is only the userspace tool required for this update or should I build
also the kernel module too ?

Is there any means that would prevent the "mdadm --scan ..." usually
on ramdisks or init scripts for touching my array ? (i.e changing the
partition types, for example ? )

>>> It wouldn't hurt to also post the "smartctl -x" for each of these drives.
>>>
>> http://pastie.org/5895385 (sdg - the really broken one - will RMA this
>> one after recovering or giving up)
>
> It doesn't appear to be broken.  Just some pending sectors that'll
> probably be cleaned up by a wipe, and would have been taken care of with
> regular scrubbing.
>
>> http://pastie.org/5895387 (sdh - apparently clean)
>> http://pastie.org/5895388 (sdi - apparently clean)
>> http://pastie.org/5895389 (sdj - apparently clean)
>
> These do show one critical piece of information that is probably the
> only real problem in your system:
>
>> Warning: device does not support SCT Error Recovery Control command
>
> You are using cheap desktop drives that do not support time limits on
> error recovery.  They are completely *unsafe* to use "out-of-the-box" in
> *any* raid array.
>
> If they did support SCTERC, you could use a boot script to set short
> timeouts.  Since they don't, your only option is a boot script to set
> very long timeouts in the linux driver for each disk.
>

I'm using WD Caviar Green disks, which are "cheap desktop drives" :).
It is a home setup after all :( . I did got some WD "Red" series which
supposedly have a "NAS friendly" firmware. Will gladly report back if
those support SCTERC. They are less than 10% more expensive nowadays
than the "Green" series.

>> #! /bin/bash
>> # Place in rc.local or wherever your distro expects boot-time scripts
>> #
>> for x in sdg sdh sdi sdj
>> do
>>     echo 180 >/sys/block/$x/device/timeout
>> done
>

Will write down this one.

> Long timeouts can have negative consequences for services that might be
> using the array, but you have no choice.  If you don't do this, any
> unrecoverable read error will cause the offending disk to be kicked out
> instead of fixed.  (Including errors found during scrubbing.)
>
>> Thanks for stepping up for help :). I did use pastie.org to avoid a
>> wall of text. some of those outputs are even bigger what is allowed by
>> pastie. Let me know if you would prefer next outputs to be inline.
>
> Yes.
>
> HTH,
>
> Phil

Once all this is solved, I would be more than happy to submit changes
to the current wiki page containing the additional information you
have been providing me that doesn't exists there, including pushing
the timeout to a long one.

Cheers,

Theo

Re: help recovering a software raid5 device

[Phil Turmel]

On 01/28/2013 09:33 AM, Theo Cabrerizo Diem wrote:

[trim /]

> Is there a way that I could flag the raid device (or the partitions)
> to not be auto-detected on boot ? I'm afraid that since the "mdadm
> --create --assume-clean" completed successfuly before, a reboot on
> this machine might bring the array fully online and, for example,
> might trigger a check or resync of data. That would be the worse case.

You would modify your mdadm.conf file per its man-page to not allow
automatic assembly, then update your initramfs.  Both of these tasks
vary by distro.

[trim /]

>> So your copy of mdadm is very new, and has the new defaults for data
>> offset (leaving more room for a bad block log).  You need to boot with a
>> slightly older liveCD or other rescue media to get a copy of mdadm that
>> is about 1 year old.  Re-run the "mdadm --create --assume-clean" with
>> that version of mdadm.
>>
>> (The development version of mdadm has command-line syntax to set the
>> data offset per device, but I don't believe it has been released yet.
>> If you are comfortable using git and compiling your own utility, that
>> would be another option.)
>>
> I have no problem compiling the tools myself. I would actually prefer
> that than triggering a reboot on the machine and having unpredictable
> results from how it would be detected after the multiple attempts to
> create the device.
> 
> Is only the userspace tool required for this update or should I build
> also the kernel module too ?

Just the userspace tool.

> Is there any means that would prevent the "mdadm --scan ..." usually
> on ramdisks or init scripts for touching my array ? (i.e changing the
> partition types, for example ? )

Just the mdadm.conf file that ends up in the initramfs.

> I'm using WD Caviar Green disks, which are "cheap desktop drives" :).
> It is a home setup after all :( . I did got some WD "Red" series which
> supposedly have a "NAS friendly" firmware. Will gladly report back if
> those support SCTERC. They are less than 10% more expensive nowadays
> than the "Green" series.

Yes, the Red drives are reported to support SCTERC.  You could modify
the script to cover both cases:

> #! /bin/bash
> # Place in rc.local or wherever your distro expects boot-time scripts
> #
> for x in sdg sdh sdi sdj
> do
>     smartctl -l scterc,70,70 /dev/$x &>/dev/null || \
>     echo 180 >/sys/block/$x/device/timeout
> done

Drives that don't accept the seven-second timeout will be given a long
driver timeout.

[trim /]

> Once all this is solved, I would be more than happy to submit changes
> to the current wiki page containing the additional information you
> have been providing me that doesn't exists there, including pushing
> the timeout to a long one.

Please do.  And also strongly recommend people ask for help on this list
if they are the least bit hesitant.

Phil