Seagate black armour recovery

Seagate black armour recovery

[Kevin Wilson]

Good day All,

I have a current problem for which I would love some advice regarding
mdadm and raid superblock information. Suffice to say that we have
been busy for a while with the recovery of a 4 disk raid pack that was
installed in a Seagate Black Armour. It seems from various posts to be
a common issue, but the 4th drive has after numerous power failures
decided to show up as a 4GB drive instead of the 2TB it should be.

Although Seagate will assist with replacement, recovery is not
supported. They suggested transposing the disks into an HP micro
server and using Ubuntu to recover the raid config and backup the
data, however they do not officially support this process.

We have done just that and although the utility seemed to sync the
timestamps and event counts of the first three disks, it will not
assemble the raid as disk 2 says disk1 is faulty and disk three says
disk 1 & 2 are faulty. We tried the --update summaries but the return
message indicated this was not supported on this version of the
superblock information.

My research on the internet has pointed to the following solutions:

1. Hexedit the drive status information in the superblocks and set it
to what we require to assemble

2. Run the create option of mdadm with precisely the original
configuration of the pack to overwrite the superblock information

Our position currently:

4 x SATA Seagate 2Tb Hard drives in RAID 5 configuration from Seagate
Black armour NAS device which seems to be running a version of
Busybox.

RAID set failed after numerous power outages in short succession of
one another. Seagate seems to partition each drive into 4 partitions,
three small partitions for mirrored boot, root and swap, and a raid 5
set from the 4 large partitions. We can still see all 4 partitions on
the first three drives.



We prefer option 2 as hex editing anything seems very complex and the
risk of losing data may be increased.

Mdadm examine scan verbose:(please note we are only interested in /dev/md/3)
ARRAY /dev/md0 level=raid1 num-devices=4
UUID=e5c4f833:10499bfb:f0bc667a:ab5d5655
   devices=/dev/sdb1,/dev/sdc1,/dev/sda1
ARRAY /dev/md1 level=raid1 num-devices=4
UUID=dd1a58c7:b4a1d054:6ac96c1f:9e4e75f3
   devices=/dev/sdb2,/dev/sdc2,/dev/sda2
ARRAY /dev/md0 level=raid1 num-devices=4
UUID=88c4cd7b:2f2c33f9:98147519:0af53b03
   devices=/dev/sdb3,/dev/sdc3,/dev/sda3
ARRAY /dev/md/3 level=raid5 metadata=1.2 num-devices=4
UUID=32dded91:8cfe88ca:765df125:42fef71b name=3
   devices=/dev/sdb4,/dev/sdc4,/dev/sda4

Mdadm examine for each drive:
/dev/sda4:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 32dded91:8cfe88ca:765df125:42fef71b
           Name : 3
  Creation Time : Wed Jun 29 18:19:40 2011
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3901415744 (1860.34 GiB 1997.52 GB)
     Array Size : 11704247040 (5581.02 GiB 5992.57 GB)
  Used Dev Size : 3901415680 (1860.34 GiB 1997.52 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : bfb2c355:6c62cbca:027b51ae:efa6d0cf

    Update Time : Mon Oct 14 08:22:08 2013
       Checksum : 57bae206 - correct
         Events : 18538

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 0
   Array State : AAA. ('A' == active, '.' == missing)

/dev/sdb4:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 32dded91:8cfe88ca:765df125:42fef71b
           Name : 3
  Creation Time : Wed Jun 29 18:19:40 2011
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3901415744 (1860.34 GiB 1997.52 GB)
     Array Size : 11704247040 (5581.02 GiB 5992.57 GB)
  Used Dev Size : 3901415680 (1860.34 GiB 1997.52 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : b56364ca:41cc4eef:f1df5dc3:2f8ca60f

    Update Time : Mon Oct 14 08:23:06 2013
       Checksum : 45bf4737 - correct
         Events : 18538

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 1
   Array State : .AA. ('A' == active, '.' == missing)

/dev/sdc4:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 32dded91:8cfe88ca:765df125:42fef71b
           Name : 3
  Creation Time : Wed Jun 29 18:19:40 2011
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3901415744 (1860.34 GiB 1997.52 GB)
     Array Size : 11704247040 (5581.02 GiB 5992.57 GB)
  Used Dev Size : 3901415680 (1860.34 GiB 1997.52 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 762c0464:cb7baf42:590e39ba:8685f4a8

    Update Time : Mon Oct 14 08:24:13 2013
       Checksum : c2e5e785 - correct
         Events : 18538

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 2
   Array State : ..A. ('A' == active, '.' == missing)

/dev/sdd4 is the faulty drive that now shows up as 4GB.

I presume we are looking for the right way to get the Array State
corrected or ignored on the assemble.

regards,

Kevin

Re: Seagate black armour recovery

[Phil Turmel]

Hi Kevin,

On 11/04/2013 08:51 AM, Kevin Wilson wrote:
> Good day All,

[snip /]

Good report, BTW.

> 1. Hexedit the drive status information in the superblocks and set it
> to what we require to assemble

You would have to be very brave to try that, and very confident that you
complete understood the on-disk raid metadata.

> 2. Run the create option of mdadm with precisely the original
> configuration of the pack to overwrite the superblock information

This is a valid option, but should always be the *last* resort.

Your research missed the recommended *first* option:

mdadm --assemble --force ....

[snip /]

> Mdadm examine for each drive:
> /dev/sda4:

>          Events : 18538
>    Device Role : Active device 0
>    Array State : AAA. ('A' == active, '.' == missing)

> /dev/sdb4:
>          Events : 18538
>    Device Role : Active device 1
>    Array State : .AA. ('A' == active, '.' == missing)

> /dev/sdc4:
>          Events : 18538
>    Device Role : Active device 2
>    Array State : ..A. ('A' == active, '.' == missing)

> /dev/sdd4 is the faulty drive that now shows up as 4GB.

Check /proc/mdstat and then use mdadm --stop to make sure any partial
assembly of these devices is gone.  Then

mdadm -Afv /dev/md3 /dev/sd[abc]4

Save the output so you can report it to this list if it fails.  You
should end up with the array running in degraded mode.

Use fsck as needed to deal with the detritus from the power losses, then
make your backups.

HTH,

Phil

Re: Seagate black armour recovery

[Kevin Wilson]

Hi Phil,
Thanks for the quick reply. I should have, as you correctly stated,
included the result from trying to force assemble.
mdadm: looking for devices for /dev/md3
mdadm: /dev/sda4 is identified as a member of /dev/md3, slot 0.
mdadm: /dev/sdb4 is identified as a member of /dev/md3, slot 1.
mdadm: /dev/sdc4 is identified as a member of /dev/md3, slot 2.
mdadm: ignoring /dev/sdb4 as it reports /dev/sda4 as failed
mdadm: ignoring /dev/sdc4 as it reports /dev/sda4 as failed
mdadm: no uptodate device for slot 1 of /dev/md3
mdadm: no uptodate device for slot 2 of /dev/md3
mdadm: no uptodate device for slot 3 of /dev/md3
mdadm: added /dev/sda4 to /dev/md3 as 0
mdadm: /dev/md3 assembled from 1 drive - not enough to start the array.

I was then trying to edit the Array status in sdb4 and sdc4 due to the
two lines ignoring /dev/sd[x]4 as it reports...
The man pages suggest using the --update=summaries with a list of the
devices, however I get an error that states that this is not valid for
1.X superblock versions.

At this point we found only the two options I mentioned, and we
decided to climb the mountain and talk to the oracle. Is there another
way to get the other two drives back into the array?

regards,

Kevin

On 5 November 2013 00:28, Phil Turmel <philip@turmel.org> wrote:
> Hi Kevin,
>
> On 11/04/2013 08:51 AM, Kevin Wilson wrote:
>> Good day All,
>
> [snip /]
>
> Good report, BTW.
>
>> 1. Hexedit the drive status information in the superblocks and set it
>> to what we require to assemble
>
> You would have to be very brave to try that, and very confident that you
> complete understood the on-disk raid metadata.
>
>> 2. Run the create option of mdadm with precisely the original
>> configuration of the pack to overwrite the superblock information
>
> This is a valid option, but should always be the *last* resort.
>
> Your research missed the recommended *first* option:
>
> mdadm --assemble --force ....
>
> [snip /]
>
>> Mdadm examine for each drive:
>> /dev/sda4:
>
>>          Events : 18538
>>    Device Role : Active device 0
>>    Array State : AAA. ('A' == active, '.' == missing)
>
>> /dev/sdb4:
>>          Events : 18538
>>    Device Role : Active device 1
>>    Array State : .AA. ('A' == active, '.' == missing)
>
>> /dev/sdc4:
>>          Events : 18538
>>    Device Role : Active device 2
>>    Array State : ..A. ('A' == active, '.' == missing)
>
>> /dev/sdd4 is the faulty drive that now shows up as 4GB.
>
> Check /proc/mdstat and then use mdadm --stop to make sure any partial
> assembly of these devices is gone.  Then
>
> mdadm -Afv /dev/md3 /dev/sd[abc]4
>
> Save the output so you can report it to this list if it fails.  You
> should end up with the array running in degraded mode.
>
> Use fsck as needed to deal with the detritus from the power losses, then
> make your backups.
>
> HTH,
>
> Phil
>

Re: Seagate black armour recovery

[Doug Ledford]

On 11/05/2013 02:39 PM, Kevin Wilson wrote:
> Hi Phil,
> Thanks for the quick reply. I should have, as you correctly stated,
> included the result from trying to force assemble.
> mdadm: looking for devices for /dev/md3
> mdadm: /dev/sda4 is identified as a member of /dev/md3, slot 0.
> mdadm: /dev/sdb4 is identified as a member of /dev/md3, slot 1.
> mdadm: /dev/sdc4 is identified as a member of /dev/md3, slot 2.
> mdadm: ignoring /dev/sdb4 as it reports /dev/sda4 as failed
> mdadm: ignoring /dev/sdc4 as it reports /dev/sda4 as failed
> mdadm: no uptodate device for slot 1 of /dev/md3
> mdadm: no uptodate device for slot 2 of /dev/md3
> mdadm: no uptodate device for slot 3 of /dev/md3
> mdadm: added /dev/sda4 to /dev/md3 as 0
> mdadm: /dev/md3 assembled from 1 drive - not enough to start the array.
>
> I was then trying to edit the Array status in sdb4 and sdc4 due to the
> two lines ignoring /dev/sd[x]4 as it reports...
> The man pages suggest using the --update=summaries with a list of the
> devices, however I get an error that states that this is not valid for
> 1.X superblock versions.

Hmmm...this looks like a legitimate bug in the raid superblock update 
code.  I'm putting Neil on the Cc: of this email so he doesn't 
accidentally overlook this issue.

So, as I see it, the bug (which is present in your mdadm -E output 
below, and confirmed in the dmesg output above) is that at some point in 
time, /dev/sdd4 failed, resulting in a superblock update on sda4, sdb4, 
and sdc4.  From the looks of it, the update landed on sda4 before 
something else happened causing the raid subsystem to mark sda4 as bad. 
  Then, we marked sda4 bad in our internal superblock and wrote that to 
sdb4, and then that must have returned a failure before we even 
attempted to write sdc4 and we marked sdb4 bad before we did.

This is what I think normally happens when we have a drive fail, but the 
rest of the system is ok:

drive X fails ->
   update event count and mark drive bad in superblock ->
     submit write to new superblock on drive A
     submit write to new superblock on drive B
     submit write to new superblock on drive C
     (delay for drive access time)
     write to new superblock on drive A completes
     write to new superblock on drive B completes
     write to new superblock on drive C completes
   superblock update complete, array in consistent, degraded state

Now, here's where I think the problem may creep in:

drive X fails ->
   update event count and mark drive bad in superblock ->
     submit write to new superblock on drive A
       write to drive A immediately fails, mark drive A bad in superblock
       but because we are in the process of doing a superblock update
       with a new event count, don't bother to increment event count
     submit write to new superblock on drive B with drive A marked bad ->
       write to drive B immediately fails, mark drive B bad in superblock
       but because we are in the process of doing a superblock update
       with a new event count, don't bother to increment event count
     submit write to new superblock on drive C, ditto on the rest
   superblock update more or less fails, but for some reason, the writes
   actually completed to disk (an interrupt issue on the
   controller would cause the write to complete but never get
   acknowledged by the disk layer, resulting in the sort of thing we see
   here, although that wouldn't explain the ordering)

I haven't actually read through the code, but this is the sort of thing 
that seems to be happening.  I don't have a better explanation for why 
the superblocks got into the state that they are.

Now, as for what to do, I think the only thing to do now is to recreate 
the array using the same information that you currently have.

Use the output of mdadm -E on a constituent device to get all the 
settings you need (save them off).  Then you should be able to get the 
superblock version, the chunk size, the presence or absence of a bitmap, 
bitmap chunk, and the data offset from the mdadm -E output you saved. 
As long as any attempts to remake the array use the same superblock 
version, use --assume-clean, keep the drives in the right order, and the 
array is created/assembled in read-only state and you just do a 
read-only fsck, then you won't corrupt anything in the array if the rest 
of the parameters aren't perfect and you can try again as many times as 
needed to get things right and get the disks back online.  The one thing 
you might have to do is track down the same version of mdadm that was 
used to create the array as the default data offset for some of the 
superblock versions has changed over time and you might not be able to 
get the data offset right without having the older mdadm version on hand.

> At this point we found only the two options I mentioned, and we
> decided to climb the mountain and talk to the oracle. Is there another
> way to get the other two drives back into the array?
>
> regards,
>
> Kevin
>
> On 5 November 2013 00:28, Phil Turmel <philip@turmel.org> wrote:
>> Hi Kevin,
>>
>> On 11/04/2013 08:51 AM, Kevin Wilson wrote:
>>> Good day All,
>>
>> [snip /]
>>
>> Good report, BTW.
>>
>>> 1. Hexedit the drive status information in the superblocks and set it
>>> to what we require to assemble
>>
>> You would have to be very brave to try that, and very confident that you
>> complete understood the on-disk raid metadata.
>>
>>> 2. Run the create option of mdadm with precisely the original
>>> configuration of the pack to overwrite the superblock information
>>
>> This is a valid option, but should always be the *last* resort.
>>
>> Your research missed the recommended *first* option:
>>
>> mdadm --assemble --force ....
>>
>> [snip /]
>>
>>> Mdadm examine for each drive:
>>> /dev/sda4:
>>
>>>           Events : 18538
>>>     Device Role : Active device 0
>>>     Array State : AAA. ('A' == active, '.' == missing)
>>
>>> /dev/sdb4:
>>>           Events : 18538
>>>     Device Role : Active device 1
>>>     Array State : .AA. ('A' == active, '.' == missing)
>>
>>> /dev/sdc4:
>>>           Events : 18538
>>>     Device Role : Active device 2
>>>     Array State : ..A. ('A' == active, '.' == missing)
>>
>>> /dev/sdd4 is the faulty drive that now shows up as 4GB.
>>
>> Check /proc/mdstat and then use mdadm --stop to make sure any partial
>> assembly of these devices is gone.  Then
>>
>> mdadm -Afv /dev/md3 /dev/sd[abc]4
>>
>> Save the output so you can report it to this list if it fails.  You
>> should end up with the array running in degraded mode.
>>
>> Use fsck as needed to deal with the detritus from the power losses, then
>> make your backups.
>>
>> HTH,
>>
>> Phil
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>