Failed Raid 5 due to OS mbr written on one of the array drives

Failed Raid 5 due to OS mbr written on one of the array drives

[Nicolas Tellier]

I'm looking for advices regarding a failed (2 disc down) Raid 5 array
with 4 disks. It was running in a NAS for quite some time but after I
came back from a trip, I found out that the system disk was dead.
After replacing the drive, I reinstalled the OS (OpenMediaVault for
the curious). Sadly the mbr was written to one of the raid disk
instead of the OS one. This would not have been too critical if, after
booting the system, I didn't realized that the array was already
running in degraded mode prior to the OS disk problem. Luckily I have
a backup for most of the critical data on that array. There is nothing
that I cannot replace, but it would still be quite inconvenient. I
guess it's a good opportunity to learn more about raid :)

 mdadm --stop /dev/md0 and removing the boot flag from the wrongly
written raid disk are the only stuff that I did so far.

Here are the informations I gathered :


root@NAStradamus:~# uname -a
Linux NAStradamus 3.2.0-4-amd64 #1 SMP Debian 3.2.68-1+deb7u5 x86_64 GNU/Linux


root@NAStradamus:~# mdadm --examine /dev/sd[a-z]1

mdadm: No md superblock detected on /dev/sda1.

/dev/sdb1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : a08bcee5:9fb42352:319ecab9:53d6277b
           Name : ArchliNAS:0
  Creation Time : Sun Jul  8 17:59:49 2012
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 1953005569 (931.27 GiB 999.94 GB)
     Array Size : 2929507584 (2793.80 GiB 2999.82 GB)
  Used Dev Size : 1953005056 (931.27 GiB 999.94 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 9b28a04c:f2c3d6c9:6f76859d:624927f0

    Update Time : Wed Sep 16 19:32:12 2015
       Checksum : 4fb98985 - correct
         Events : 108016

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 1
   Array State : AAA. ('A' == active, '.' == missing)

/dev/sdc1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : a08bcee5:9fb42352:319ecab9:53d6277b
           Name : ArchliNAS:0
  Creation Time : Sun Jul  8 17:59:49 2012
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 1953005569 (931.27 GiB 999.94 GB)
     Array Size : 2929507584 (2793.80 GiB 2999.82 GB)
  Used Dev Size : 1953005056 (931.27 GiB 999.94 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 1952e898:50043e66:8247a64d:72ffb6c0

    Update Time : Wed Sep 16 19:32:12 2015
       Checksum : b9197a85 - correct
         Events : 108016

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 2
   Array State : AAA. ('A' == active, '.' == missing)


root@NAStradamus:~# mdadm --examine /dev/sdd
/dev/sdd:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : a08bcee5:9fb42352:319ecab9:53d6277b
           Name : ArchliNAS:0
  Creation Time : Sun Jul  8 17:59:49 2012
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 1953523120 (931.51 GiB 1000.20 GB)
     Array Size : 2929507584 (2793.80 GiB 2999.82 GB)
  Used Dev Size : 1953005056 (931.27 GiB 999.94 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 425c85ec:0c038b4b:cd59b4b5:280bf233

    Update Time : Mon May  4 08:18:04 2015
       Checksum : cca22997 - correct
         Events : 55532

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 3
   Array State : AAAA ('A' == active, '.' == missing)


root@NAStradamus:~# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : inactive sdb1[1] sdc1[3]
      1953005569 blocks super 1.2

unused devices: <none>


root@NAStradamus:~# mdadm --examine --scan
ARRAY /dev/md/0 metadata=1.2 UUID=a08bcee5:9fb42352:319ecab9:53d6277b
name=ArchliNAS:0


root@NAStradamus:~# fdisk -l

Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
107 heads, 58 sectors/track, 314780 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00048269

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1            2048  1953269760   976633856+  da  Non-FS data

Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048  1953269760   976633856+  da  Non-FS data

Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1            2048  1953269760   976633856+  da  Non-FS data

Disk /dev/sdd: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/sdd doesn't contain a valid partition table





First off we can see that /dev/sdd is in a weird state. You need to
know that it was added later in order to grow the array. It seems that
I was drunk when I did that as I didn't put any partition on that
disk…
Besides the event count is way off. My understanding is that there is
nothing to be done here, apart from re-adding it to the array once it
is up and running again (hopefully) and once the disk has been checked
for error and reinitialized (with a proper partition this time !).

Now, /dev/sda is more interesting. The partition is still present and
looks intact, it seems like it's just missing the superblock because
of the mbr shenanigan. Also the two healthy drives still see it as
active.

After looking around on the internet, I found people suggesting to
re-create the raid. It seems a bit extreme to me, but I cannot find
any other solution…
Luckily I saved the original command used to create this array. Here
is the one I think would be relevant in this case :

mdadm --create --verbose --assume-clean /dev/md0 --level=5
--metadata=1.2 --chunk=128 --raid-devices=4 /dev/sda1 /dev/sdb1
/dev/sdc1 missing /dev/sdd

This would be followed by a backup, the re-addition of /dev/sdd and a
migration to raid 6 with 2 more disks.
The wiki advise to get an experienced person review the measures
you're about to take, I don't know anybody experienced in RAID, hence
this e-mail :) What do you think ?

Please, CC me on the answers/comments posted to the list in response
to this; I'm not subscribed to the mailing list.

Thanks in advance for your time !

Nico
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Failed Raid 5 due to OS mbr written on one of the array drives

[Phil Turmel]

Good afternoon Nicolas,

On 11/02/2015 12:02 PM, Nicolas Tellier wrote:
> I'm looking for advices regarding a failed (2 disc down) Raid 5 array
> with 4 disks. It was running in a NAS for quite some time but after I
> came back from a trip, I found out that the system disk was dead.
> After replacing the drive, I reinstalled the OS (OpenMediaVault for
> the curious). Sadly the mbr was written to one of the raid disk
> instead of the OS one. This would not have been too critical if, after
> booting the system, I didn't realized that the array was already
> running in degraded mode prior to the OS disk problem. Luckily I have
> a backup for most of the critical data on that array. There is nothing
> that I cannot replace, but it would still be quite inconvenient. I
> guess it's a good opportunity to learn more about raid :)

Very good.  Many people don't understand that RAID != Backup.

>  mdadm --stop /dev/md0 and removing the boot flag from the wrongly
> written raid disk are the only stuff that I did so far.

Good.

[trim /]

> Now, /dev/sda is more interesting. The partition is still present and
> looks intact, it seems like it's just missing the superblock because
> of the mbr shenanigan. Also the two healthy drives still see it as
> active.

The superblock isn't anywhere near the MBR, so something more than just
grub-install must have happened.  Be prepared to have lost more of
/dev/sda1 than you've yet discovered.

> After looking around on the internet, I found people suggesting to
> re-create the raid. It seems a bit extreme to me, but I cannot find
> any other solution…

Unfortunately, yes.  You are past the point of a forced assembly or
other normal operations.

> Luckily I saved the original command used to create this array. Here
> is the one I think would be relevant in this case :
> 
> mdadm --create --verbose --assume-clean /dev/md0 --level=5
> --metadata=1.2 --chunk=128 --raid-devices=4 /dev/sda1 /dev/sdb1
> /dev/sdc1 missing /dev/sdd

1) Leave off /dev/sdd
2) Include --data-offset=262144

If it runs and you can access the filesystem (I suspect not), set up a
partition on sdd and add that to your array.  Use zero-superblock to
blow away the stale superblock on sdd.

If it doesn't work due to more sda1 damage and you end up starting over,
consider wiping all of those partition tables and creating the new array
on all bare devices.  That'll minimize the chance of a misidentified
device later.

You'll want to investigate the health of sdd.  If it's healthy, then its
drop-out must have been for some other reason.  You'll want to ensure
that doesn't happen again.

Consider browsing the archives and/or subscribing to learn the many
pitfalls for the unwary (hint: try searching for "timeout mismatch").

Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Failed Raid 5 due to OS mbr written on one of the array drives

[Nicolas Tellier]

Hi Phil, and thanks for taking the time to reply to me.

I'm not sure I understood you advice correctly.
When you say leave off /dev/sdd, do you mean I should recreate a 3
disks array like this :
mdadm --create --verbose --assume-clean /dev/md0 --level=5
--metadata=1.2 --chunk=128 --data-offset=262144 --raid-devices=3
/dev/sda1 /dev/sdb1 /dev/sdc1

Or a 4 disks array with no mention of /dev/sdd, like this (my instinct
tell me that wouldn't work) :
mdadm --create --verbose --assume-clean /dev/md0 --level=5
--metadata=1.2 --chunk=128 --data-offset=262144 --raid-devices=4
/dev/sda1 /dev/sdb1 /dev/sdc1

Or, as I was thinking originally, to put /dev/sdd as missing, like so :
mdadm --create --verbose --assume-clean /dev/md0 --level=5
--metadata=1.2 --chunk=128 --data-offset=262144 --raid-devices=4
/dev/sda1 /dev/sdb1 /dev/sdc1 missing /dev/sdd


In the eventuality of having to start back from scratch, I'll be sure
to clean everything (zero-superblock and all) and check the health of
all the devices. Raid 6 might be more suited to withstand some of my
episodic stupidity :)

I'm tempted to subscribe to the mailing list, but I'm afraid of the
volume of e-mail I'm going to get. I originally tried to search
through the archives, but couldn't find anything relevant to my case.
I'm looking right now at the "timeout mismatch" results.

Thanks again.

Nico

2015-11-02 19:19 GMT+01:00 Phil Turmel <philip@turmel.org>:
> Good afternoon Nicolas,
>
> On 11/02/2015 12:02 PM, Nicolas Tellier wrote:
>> I'm looking for advices regarding a failed (2 disc down) Raid 5 array
>> with 4 disks. It was running in a NAS for quite some time but after I
>> came back from a trip, I found out that the system disk was dead.
>> After replacing the drive, I reinstalled the OS (OpenMediaVault for
>> the curious). Sadly the mbr was written to one of the raid disk
>> instead of the OS one. This would not have been too critical if, after
>> booting the system, I didn't realized that the array was already
>> running in degraded mode prior to the OS disk problem. Luckily I have
>> a backup for most of the critical data on that array. There is nothing
>> that I cannot replace, but it would still be quite inconvenient. I
>> guess it's a good opportunity to learn more about raid :)
>
> Very good.  Many people don't understand that RAID != Backup.
>
>>  mdadm --stop /dev/md0 and removing the boot flag from the wrongly
>> written raid disk are the only stuff that I did so far.
>
> Good.
>
> [trim /]
>
>> Now, /dev/sda is more interesting. The partition is still present and
>> looks intact, it seems like it's just missing the superblock because
>> of the mbr shenanigan. Also the two healthy drives still see it as
>> active.
>
> The superblock isn't anywhere near the MBR, so something more than just
> grub-install must have happened.  Be prepared to have lost more of
> /dev/sda1 than you've yet discovered.
>
>> After looking around on the internet, I found people suggesting to
>> re-create the raid. It seems a bit extreme to me, but I cannot find
>> any other solution…
>
> Unfortunately, yes.  You are past the point of a forced assembly or
> other normal operations.
>
>> Luckily I saved the original command used to create this array. Here
>> is the one I think would be relevant in this case :
>>
>> mdadm --create --verbose --assume-clean /dev/md0 --level=5
>> --metadata=1.2 --chunk=128 --raid-devices=4 /dev/sda1 /dev/sdb1
>> /dev/sdc1 missing /dev/sdd
>
> 1) Leave off /dev/sdd
> 2) Include --data-offset=262144
>
> If it runs and you can access the filesystem (I suspect not), set up a
> partition on sdd and add that to your array.  Use zero-superblock to
> blow away the stale superblock on sdd.
>
> If it doesn't work due to more sda1 damage and you end up starting over,
> consider wiping all of those partition tables and creating the new array
> on all bare devices.  That'll minimize the chance of a misidentified
> device later.
>
> You'll want to investigate the health of sdd.  If it's healthy, then its
> drop-out must have been for some other reason.  You'll want to ensure
> that doesn't happen again.
>
> Consider browsing the archives and/or subscribing to learn the many
> pitfalls for the unwary (hint: try searching for "timeout mismatch").
>
> Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Failed Raid 5 due to OS mbr written on one of the array drives

[Adam Goryachev]

On 04/11/15 01:21, Nicolas Tellier wrote:
> Hi Phil, and thanks for taking the time to reply to me.
>
> I'm not sure I understood you advice correctly.
> When you say leave off /dev/sdd, do you mean I should recreate a 3
> disks array like this :
> mdadm --create --verbose --assume-clean /dev/md0 --level=5
> --metadata=1.2 --chunk=128 --data-offset=262144 --raid-devices=3
> /dev/sda1 /dev/sdb1 /dev/sdc1
No, but lucky you checked
> Or a 4 disks array with no mention of /dev/sdd, like this (my instinct
> tell me that wouldn't work) :
> mdadm --create --verbose --assume-clean /dev/md0 --level=5
> --metadata=1.2 --chunk=128 --data-offset=262144 --raid-devices=4
> /dev/sda1 /dev/sdb1 /dev/sdc1
No, but lucky you checked
> Or, as I was thinking originally, to put /dev/sdd as missing, like so :
> mdadm --create --verbose --assume-clean /dev/md0 --level=5
> --metadata=1.2 --chunk=128 --data-offset=262144 --raid-devices=4
> /dev/sda1 /dev/sdb1 /dev/sdc1 missing /dev/sdd
No, but lucky you checked

What he means (I'm pretty sure) is that you should run the command:

mdadm --create --verbose --assume-clean /dev/md0 --level=5
--metadata=1.2 --chunk=128 --data-offset=262144 --raid-devices=4
/dev/sda1 /dev/sdb1 /dev/sdc1 missing /dev/sdd

but delete "/dev/sdd" from that, because you are using the wrong syntax.
Specifically, run this command:

mdadm --create --verbose --assume-clean /dev/md0 --level=5
--metadata=1.2 --chunk=128 --data-offset=262144 --raid-devices=4
/dev/sda1 /dev/sdb1 /dev/sdc1 missing

The word missing replaces a device name, because the device doesn't
exist. It doesn't say that the next device name is missing.


> I'm tempted to subscribe to the mailing list, but I'm afraid of the
> volume of e-mail I'm going to get. I originally tried to search
> through the archives, but couldn't find anything relevant to my case.
> I'm looking right now at the "timeout mismatch" results.
The list isn't that busy, I would guess around 10 per day. You can
always unsubscribe after you solve your problem, but staying subscribed
will let you learn a lot about common issues, especially the drive
timeout mismatch issues.

Regards,
Adam

[SNIP]

-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

Re: Failed Raid 5 due to OS mbr written on one of the array drives

[Phil Turmel]

Hi Nicolas,

{ Top posting repaired.  Please don't do that.  Convention on kernel.org
is reply-to-all, trim unnecessary quotes, and either bottom post or
interleave. }

On 11/03/2015 09:21 AM, Nicolas Tellier wrote:
> 2015-11-02 19:19 GMT+01:00 Phil Turmel <philip@turmel.org>:
>> Good afternoon Nicolas,

[trim /]

>>> Luckily I saved the original command used to create this array. Here
>>> is the one I think would be relevant in this case :
>>>
>>> mdadm --create --verbose --assume-clean /dev/md0 --level=5
>>> --metadata=1.2 --chunk=128 --raid-devices=4 /dev/sda1 /dev/sdb1
>>> /dev/sdc1 missing /dev/sdd
>>
>> 1) Leave off /dev/sdd
>> 2) Include --data-offset=262144

> Hi Phil, and thanks for taking the time to reply to me.
>
> I'm not sure I understood you advice correctly.
> When you say leave off /dev/sdd, do you mean I should recreate a 3
> disks array like this :
> mdadm --create --verbose --assume-clean /dev/md0 --level=5
> --metadata=1.2 --chunk=128 --data-offset=262144 --raid-devices=3
> /dev/sda1 /dev/sdb1 /dev/sdc1

No.

> Or a 4 disks array with no mention of /dev/sdd, like this (my instinct
> tell me that wouldn't work) :
> mdadm --create --verbose --assume-clean /dev/md0 --level=5
> --metadata=1.2 --chunk=128 --data-offset=262144 --raid-devices=4
> /dev/sda1 /dev/sdb1 /dev/sdc1

No, but closer.

> Or, as I was thinking originally, to put /dev/sdd as missing, like so:
> mdadm --create --verbose --assume-clean /dev/md0 --level=5
> --metadata=1.2 --chunk=128 --data-offset=262144 --raid-devices=4
> /dev/sda1 /dev/sdb1 /dev/sdc1 missing /dev/sdd

No.  The keyword "missing" takes the place of a device you don't want to
include:

mdadm --create --verbose --assume-clean /dev/md0 --level=5
--metadata=1.2 --chunk=128 --data-offset=262144 --raid-devices=4
/dev/sda1 /dev/sdb1 /dev/sdc1 missing

> In the eventuality of having to start back from scratch, I'll be sure
> to clean everything (zero-superblock and all) and check the health of
> all the devices. Raid 6 might be more suited to withstand some of my
> episodic stupidity :)

I recommend raid6 for all bulk storage arrays larger than 4T capacity,
unless the data can be easily reconstructed from other sources.

> I'm tempted to subscribe to the mailing list, but I'm afraid of the
> volume of e-mail I'm going to get. I originally tried to search
> through the archives, but couldn't find anything relevant to my case.
> I'm looking right now at the "timeout mismatch" results.

This list gets a few dozen emails a day, or less.  I use filtering tools
to put the mails into a dedicated folder, skipping my inbox.  (Unless
I'm named due to reply-to-all.)  That makes them easy to manage.

Please take the timeout mismatch problem seriously -- it breaks many
people's arrays.

Phil