Re: Rebuilding a RAID5 array after drive (hardware) failure

[NeilBrown <neilb@suse.de>]

[-- Attachment #1: Type: text/plain, Size: 4703 bytes --]

On Thu, 22 May 2014 06:31:58 +0200 George Duffield
<forumscollective@gmail.com> wrote:

> I have a RAID5 array comprised of 4 x 3TB Seagate 7200 RPM SATAII
> drives.    The array was created on Ubuntu Server running on a HP
> Microserver N54L using the following command:
> 
> sudo mdadm --create --verbose /dev/md0 --raid-devices=4 --level=5
> /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
> 
> Formatted using:
> mkfs.ext4 -b 4096 -E stride=128,stripe-width=384 /dev/md0
> 
> The array is mounted in /etc/fstab by reference to its UUID and is now
> near full.
> 
> A few days back I turned on the server to access some of the files
> stored on it when I found the server was not present on the network.
> Inspecting the actual server (connected kb & monitor) I noticed that
> the machine had not progressed beyond the BIOS post screen – one of
> the drives had become damaged (2nd drive in same slot in same
> Microserver to be damaged the same way – drive spins up fine, machine
> knows it's there, but can't communicat successfully with the drive).
> In any event, suffice it to say the drive is history – it and the
> Microserver will be RMAd when this is over.
> 
> So, I'm now left with a degraded array comprising 3x3TB drives. I've
> purchased a replacement drive (same make and model) in the interim
> (and I've yet to boot this machine with the old drive removed or the
> new one inserted i.e. from an OS standpoint Ubuntu/mdadm does not yet
> know the array is degraded).
> 
> As I've lost complete faith in the Microserver (and it may very well
> damage the new drive during recovery of the array) I've also purchased
> and assembled a 2nd machine with 6 on board SATA ports rather than
> rely on another Microserver.  My intention is to remove the drives
> from the Microserver and install them in the new machine (which I'll
> boot off the same USB flash drive I used to boot the Microserver from
> [to further complicate things it seems my flash drive may also be
> corrupted, so I may have to recover from a fresh Ubuntu install and
> reassemble the array]).
> 
> A few questions if I may:
> - Is moving the array to another computer and recovering it on the new
> computer running Ubuntu Server likely to present any particular
> challenges?

No.  If you were trying to boot of the array that you moved it might be
interesting.  But as you aren't I cannot see any possible issue (assuming the
hardware functions correctly).

> 
> - Does the order/ sequence of connection of the drives to the
> motherboard matter?

No.

> 
> Another way of asking the aforementioned question is whether mdadm
> would care if one swapped drives in Microserver backplane/ PC SATA
> ports such that the physical backplane slot/ SATA port that one/more
> of the drives occupies differs from that it occupied when the array
> was created?

No.  mdadm looks at the content of the devices, not their location.


> 
> - How would I best approach rebuilding the array, my current thinking
> is as follows:
> = Identify with certainty which drive has failed - this will be done
> by removing the OS flash drive from the Microserver and disconnecting
> all drives from the backplane other than the one I believe is faulty
> (first slot on backplane) and booting the machine.  The failed drive
> causes a POST failure and is thus easily identified.
> = Remove all drives from the Microserver and install into new PC
> referenced above, at the same time replacing the failed drive with the
> replacement I purchased
> = Powering new PC via UPS
> = Booting the PC from the flash drive
> = Allowing the degraded array to be assembled by mdadm when prompted at boot
> = Adding the replacement drive to the array and allowing the array to
> be re-synchronized
> = If I'm not able to access the flash drive I will create a fresh
> install of Ubuntu Server and attempt to recreate the array in the
> fresh install.
> 
> All thoughts/ comments/ guidance much appreciated.

Sounds good.
Though I would discourage the boot sequence from assembling the degraded
array if possible.
Just get the machine up with the drive untouched.  Then use "mdadm -E" to
look at each device and make sure they are what you think they are (e.g.
consistent Event numbers etc).
Then
  mdadm --assemble /dev/mdWHATEVER ..list-of-devices...

Then make sure that looks good.
Then
  mdadm /dev/mdWHATEVER --add new-device

NeilBrown


> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]