Resurrecting a Dirty RAID-5

Resurrecting a Dirty RAID-5

[jtroan]

I've accidentally toasted a software RAID-5 on one of my VM hosts, taking a
small number of VMs with it.  (Fortunately, no unique data was lost as
every server has a "twinned" system or the data can be regenerated from
elsewhere via script.)

Short background --  the system is RHEL 6.5 and has a three-drive RAID-5
with most of the space.  ( /boot is a RAID-1 and it seems to be fine in a
degraded state.)   The system locked up when (presumably) one of the drives
died.  Upon reboot, I saw errors for "ata2", so I naturally forgot that the
drives started at ata0 and unplugged the drive at "SATA2" (DOH!).  The
system errored and halted its boot when the root file system was no longer
available.  I then switched around cables and brought up ata0 and ata1, but
the RAID won't start because of the dirty broken raid.

I booted from the RHEL 6.5 DVD and entered rescue mode.  Using information
from the Linux RAID wiki [1], I was able to confirm that my sd0 and sd1
drives are alive and have all three partitions I had originally deployed to
them -- /boot, /, and swap space.

Following the advice on the wiki [1], I'm asking for some help in using
mdadm via rescue mode in trying to get this dirty RAID-5 back on its feet
with the surviving two drives.  (I've got a third drive already on hand
that's I'll then use to bring the RAID back to full robustness.)

[1] https://raid.wiki.kernel.org/index.php/RAID_Recovery


Many thanks for any assistance that can be provided.....

=======================================================================
John M. Troan  <jtroan@jt-sw.com>
Maintainer: Football Site @ JT-SW.com
  http://www.jt-sw.com/football
Chief of Computer Operations
  U.S.S. Kitty Hawk / NCC-1659
=======================================================================

Re: Resurrecting a Dirty RAID-5

[Phil Turmel]

Good morning John,

On 07/08/2015 08:14 AM, jtroan@jt-sw.com wrote:

[trim /]

> I booted from the RHEL 6.5 DVD and entered rescue mode.  Using information
> from the Linux RAID wiki [1], I was able to confirm that my sd0 and sd1
> drives are alive and have all three partitions I had originally deployed to
> them -- /boot, /, and swap space.
> 
> Following the advice on the wiki [1], I'm asking for some help in using
> mdadm via rescue mode in trying to get this dirty RAID-5 back on its feet
> with the surviving two drives.  (I've got a third drive already on hand
> that's I'll then use to bring the RAID back to full robustness.)

From this report, you should only need to do a forced assembly from the
rescue environment with the good devices.  Like so (substituting actual
names):

mdadm -Afv /dev/md2 /dev/sda2 /dev/sdb2

If that fails, paste the verbose output in your reply here.

If the above succeeds, you may shut down, plug in your new drive, and
boot into your normal environment (still degraded, but should be
bootable).  Then add the new drive's partitions to each array.

Don't do *anything* else.

HTH,

Phil

Re: Resurrecting a Dirty RAID-5

[jtroan]

Many, many thanks for the solution.  I was able to run to a pair of mdadm
-A commands to stitch both RAID devices back together and got it to
completely boot up on the two drives.

As expected, it did note the degraded md1 since the third drive was still
missing.  I'll install it when I have a few more spare moments, and have
left the system powered down for now.

I'm also thinking about adding a fourth drive and try to config it as a
hot-spare, giving me some extra margin for failure.

=======================================================================
John M. Troan  <jtroan@jt-sw.com>
Maintainer: Football Site @ JT-SW.com
  http://www.jt-sw.com/football
Chief of Computer Operations
U.S.S. Kitty Hawk / NCC-1659
=======================================================================


Phil Turmel <philip@turmel.org> wrote on 07/08/2015 09:19:48:

>
> To: jtroan@jt-sw.com, linux-raid@vger.kernel.org,
> Subject: Re: Resurrecting a Dirty RAID-5
>
> Good morning John,
>
> On 07/08/2015 08:14 AM, jtroan@jt-sw.com wrote:
>
> [trim /]
>
> > I booted from the RHEL 6.5 DVD and entered rescue mode.  Using
information
> > from the Linux RAID wiki [1], I was able to confirm that my sd0 and sd1
> > drives are alive and have all three partitions I had originally
deployed to
> > them -- /boot, /, and swap space.
> >
> > Following the advice on the wiki [1], I'm asking for some help in using
> > mdadm via rescue mode in trying to get this dirty RAID-5 back on its
feet
> > with the surviving two drives.  (I've got a third drive already on hand
> > that's I'll then use to bring the RAID back to full robustness.)
>
> From this report, you should only need to do a forced assembly from the
> rescue environment with the good devices.  Like so (substituting actual
> names):
>
> mdadm -Afv /dev/md2 /dev/sda2 /dev/sdb2
>
> If that fails, paste the verbose output in your reply here.
>
> If the above succeeds, you may shut down, plug in your new drive, and
> boot into your normal environment (still degraded, but should be
> bootable).  Then add the new drive's partitions to each array.
>
> Don't do *anything* else.
>
> HTH,
>
> Phil

Re: Resurrecting a Dirty RAID-5

[Mikael Abrahamsson]

On Wed, 8 Jul 2015, jtroan@jt-sw.com wrote:

> I'm also thinking about adding a fourth drive and try to config it as a 
> hot-spare, giving me some extra margin for failure.

I would recommend you turn your raid5 into raid6 instead of having a 
raid5+spare. This can be done with a fairly simple command, without 
downtime.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: Resurrecting a Dirty RAID-5

[jtroan]

>
> On Wed, 8 Jul 2015, jtroan@jt-sw.com wrote:
>
> > I'm also thinking about adding a fourth drive and try to config it as a

> > hot-spare, giving me some extra margin for failure.
>
> I would recommend you turn your raid5 into raid6 instead of having a
> raid5+spare. This can be done with a fairly simple command, without
> downtime.
>

I like the idea of using RAID-6 for / (for all the VMs under /var).  (I
figure I'll probably still have to do a spare on my RAID-1 device
for /boot .)

What's the (mdadm?) command to convert an MD from RAID-5 to RAID-6?

Thanks.....

=======================================================================
John M. Troan  <jtroan@jt-sw.com>
Maintainer: Football Site @ JT-SW.com
  http://www.jt-sw.com/football
Chief of Computer Operations
  U.S.S. Kitty Hawk / NCC-1659
=======================================================================

Re: Resurrecting a Dirty RAID-5

[Adam Goryachev]

On 13/07/15 01:01, jtroan@jt-sw.com wrote:
>> On Wed, 8 Jul 2015, jtroan@jt-sw.com wrote:
>>
>>> I'm also thinking about adding a fourth drive and try to config it as a
>>> hot-spare, giving me some extra margin for failure.
>> I would recommend you turn your raid5 into raid6 instead of having a
>> raid5+spare. This can be done with a fairly simple command, without
>> downtime.
>>
> I like the idea of using RAID-6 for / (for all the VMs under /var).  (I
> figure I'll probably still have to do a spare on my RAID-1 device
> for /boot .)
>
> What's the (mdadm?) command to convert an MD from RAID-5 to RAID-6?
>
I think the standard option would be to have a hot spare, and then 
something like:
mdadm /dev/mdX --grow --level=raid6

Also, for your raid1, never have a hot spare, just do this:
mdadm /dev/mdX --grow --raid-devices=3
Then you will always have all your data replicated on all three drives, 
so again, no hot spare required.

Also remember to duplicate the grub/whatever boot sector....

Regards,
Adam

-- 
Adam Goryachev Website Managers www.websitemanagers.com.au

Re: Resurrecting a Dirty RAID-5

[Can Jeuleers]

On 13/07/15 00:33, Adam Goryachev wrote:
> Also, for your raid1, never have a hot spare, just do this:
> mdadm /dev/mdX --grow --raid-devices=3
> Then you will always have all your data replicated on all three drives,
> so again, no hot spare required.

Never say never, as there are valid use cases for having hot spares in a
RAID1 set.

My own use case is that I want to be reasonably assured that my spare
won't fail at around the same time as the active disks (due to having
the same age and having been subjected to exactly the same workload).

Re: Resurrecting a Dirty RAID-5

[Adam Goryachev]

On 13/07/15 16:31, Can Jeuleers wrote:
> On 13/07/15 00:33, Adam Goryachev wrote:
>> Also, for your raid1, never have a hot spare, just do this:
>> mdadm /dev/mdX --grow --raid-devices=3
>> Then you will always have all your data replicated on all three drives,
>> so again, no hot spare required.
> Never say never, as there are valid use cases for having hot spares in a
> RAID1 set.
>
> My own use case is that I want to be reasonably assured that my spare
> won't fail at around the same time as the active disks (due to having
> the same age and having been subjected to exactly the same workload).

Yes, this is true too... though with physical HDD, I would expect there 
are enough differences to mean that they will fail at different times, 
even SSD's should have enough variance.

Of course, the other option is to purchase the drives at different times 
(eg, one month apart) so they are also from different batches, as well 
as have a month or two difference in how they are used.

Of course, if both drives in the RAID1 fail at the exact same moment, 
how is a hot spare better than a three drive raid1? Wouldn't the chance 
of three drives failing at the same critical moment be less than the 
chance of two drives failing at the same time (or one drive to fail, and 
then the second to fail during the increased load of a resync)?

I really have no idea about the actual statistical numbers/chances, but 
sounds like a valid question to me...

PS, of course, you should never say never, so I do still agree with you, 
someone, somewhere might have a reason to do it differently. However, if 
they really do, then they should know better than me.

Regards,
Adam

-- 
Adam Goryachev
Website Managers
P: +61 2 8304 0000                    adam@websitemanagers.com.au
F: +61 2 8304 0001                     www.websitemanagers.com.au