Replacing a RAID1 drive that has not failed.

Replacing a RAID1 drive that has not failed.

[Doug Herr]

I think it might be time to replace one or both of my RAID 1 drives.  
They have been giving good service for six years.  Both still have good 
smartctl reports but I am ready to move one out to be used as a member of 
my backup solution using old drives to hold rsync backups.  I would thus 
give my RAID a fresh disk before there is an actual failure.

I recently experimented with adding a third partition to my swap raid 
partition and that went very well, so I was thinking that this would 
provide a safer method to replace one drive compared to failing it and 
then replacing it.

My test makes me think it might be as easy as:

 1. Partition new drive (plugged in via external SATA dock)
Use: fdisk /dev/sdc

 2. Add 3rd partition to each RAID.
Use: mdadm /dev/md4 --grow --raid-devices=3 --add /dev/sdc2

 3. Wait for all the resyncs.
Use: watch cat /proc/mdstat

 4. Fail out sda for each RAID.
Use: mdadm /dev/md4 --fail /dev/sda2

 5. Drop back down to two disk.
Use: mdadm /dev/md4 --grow --raid-devices=2

 6. Make the old partitions "safe".
Use: mdadm --zero-superblock /dev/sdc2

 7. Power down and swap sdc into sda slot.

I only tested that with version 1.2 metadata, so I would be interested if 
there is anything above that would not work with version 0.90.  Also note 
that I did my test using my swap partition but it seems like the 
procedure should not care what sort of partitioning is used.


Below is info about my setup and how I tested to come up with the above 
steps...

I am:
15:40-doug@wombat-~>uname -a
Linux wombat.wombatz.com 4.2.3-200.fc22.x86_64 #1 SMP Thu Oct 8 03:23:55 
UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

15:40-doug@wombat-~>cat /etc/fedora-release 
Fedora release 22 (Twenty Two)

15:40-doug@wombat-~>mdadm -V
mdadm - v3.3.2 - 21st August 2014

My RAID setup:

(I am not using RAID for /boot, instead I rsync /boot to /bootalt each 
night.  Otherwise everything is on RAID 1, including swap.)

15:23-doug@wombat-~/wombat-4/raid/current-2015-10-26>df -hT |grep -v 
^tmpfs
Filesystem     Type      Size  Used Avail Use% Mounted on
devtmpfs       devtmpfs  4.4G     0  4.4G   0% /dev
/dev/md126     ext4       30G   11G   18G  39% /
/dev/sdb1      ext4      579M  126M  411M  24% /bootalt
/dev/md1       ext4       40G   18G   23G  44% /home
/dev/md123     ext4      422G  209G  192G  53% /data2
/dev/md125     ext4      422G   74G  327G  19% /data1
/dev/sda1      ext4      579M  126M  411M  24% /boot

15:09-doug@wombat-~>swapon -s
Filename				Type		Size	Used	
Priority
/dev/md4                               	partition	4946980	0	-1

(I am using UUID for /etc/mdadm.conf and in /etc/fstab so I did not 
bother to "fix" the names when they got changed to md12x during Fedora 
upgrades.)


15:19-doug@wombat-~>cat /proc/mdstat 
Personalities : [raid1] 
md123 : active raid1 sdb7[1] sda7[0]
      448888128 blocks [2/2] [UU]
      bitmap: 0/4 pages [0KB], 65536KB chunk

md125 : active raid1 sdb6[1] sda6[0]
      448888128 blocks [2/2] [UU]
      bitmap: 1/4 pages [4KB], 65536KB chunk

md1 : active raid1 sdb5[0] sda5[2]
      41952620 blocks super 1.2 [2/2] [UU]
      bitmap: 1/1 pages [4KB], 65536KB chunk

md126 : active raid1 sdb3[0] sda3[1]
      31463232 blocks [2/2] [UU]
      bitmap: 1/1 pages [4KB], 65536KB chunk

md4 : active raid1 sdb2[2] sda2[3]
      4946984 blocks super 1.2 [2/2] [UU]
      
unused devices: <none>



Notes from the test I did using a partion on a older and smaller drive...

[root@wombat Documents]# mdadm --detail /dev/md4
/dev/md4:
        Version : 1.2
  Creation Time : Sat Mar 24 16:53:28 2012
     Raid Level : raid1
     Array Size : 4946984 (4.72 GiB 5.07 GB)
  Used Dev Size : 4946984 (4.72 GiB 5.07 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Fri Oct 23 10:55:06 2015
          State : clean 
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           Name : wombat.wombatz.com:4  (local to host wombat.wombatz.com)
           UUID : 2382da1b:c77c04c7:9e4f955c:9450c809
         Events : 541

    Number   Major   Minor   RaidDevice State
       3       8        2        0      active sync   /dev/sda2
       2       8       18        1      active sync   /dev/sdb2


[root@wombat Documents]# fdisk /dev/sdc
[snip]


[root@wombat doug]# mdadm /dev/md4 --grow --raid-devices=3 --add /dev/sdc2
mdadm: added /dev/sdc2
unfreeze


[root@wombat Documents]# cat /proc/mdstat 
[snip]
md4 : active raid1 sdc2[4] sdb2[2] sda2[3]
      4946984 blocks super 1.2 [3/2] [UU_]
      [=======>.............]  recovery = 39.2% (1941056/4946984) 
finish=0.6min speed=71890K/sec

Soon it showed:

md4 : active raid1 sdc2[4] sdb2[2] sda2[3]
      4946984 blocks super 1.2 [3/3] [UUU]


And now:

[root@wombat Documents]# mdadm --detail /dev/md4
/dev/md4:
        Version : 1.2
  Creation Time : Sat Mar 24 16:53:28 2012
     Raid Level : raid1
     Array Size : 4946984 (4.72 GiB 5.07 GB)
  Used Dev Size : 4946984 (4.72 GiB 5.07 GB)
   Raid Devices : 3
  Total Devices : 3
    Persistence : Superblock is persistent

    Update Time : Sat Oct 24 12:57:39 2015
          State : clean 
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

           Name : wombat.wombatz.com:4  (local to host wombat.wombatz.com)
           UUID : 2382da1b:c77c04c7:9e4f955c:9450c809
         Events : 562

    Number   Major   Minor   RaidDevice State
       3       8        2        0      active sync   /dev/sda2
       2       8       18        1      active sync   /dev/sdb2
       4       8       33        2      active sync   /dev/sdc2


Now try to remove it:
[root@wombat doug]# mdadm /dev/md4 --fail /dev/sdc2
mdadm: set /dev/sdc2 faulty in /dev/md4

[root@wombat doug]# mdadm /dev/md4 --grow --raid-devices=2 
raid_disks for /dev/md4 set to 2
unfreeze

Seems good:
md4 : active raid1 sdc2[4](F) sdb2[2] sda2[3]
      4946984 blocks super 1.2 [2/2] [UU]

Just to be safe:
mdadm --zero-superblock /dev/sdc2


-- 
Doug Herr 

Fwd: Replacing a RAID1 drive that has not failed.

[Edward Kuns]

On Mon, Oct 26, 2015 at 6:00 PM, Doug Herr <gmane@wombatz.com> wrote:
> I think it might be time to replace one or both of my RAID 1 drives. [snip]
> I recently experimented with adding a third partition to my swap raid
> partition and that went very well, so I was thinking that this would
> provide a safer method to replace one drive compared to failing it and
> then replacing it.
> [snip]

I tried basically the procedure you described, based on Internet
research, to replace a working but suspect drive in a RAID.  I ran
into problems.  FWIW, my metadata format is 1.2.  You can see the
thread at  http://marc.info/?l=linux-raid&m=143668094907891&w=2
although the most pertinent EMail is the first response
http://marc.info/?l=linux-raid&m=143670871711935&w=2

I suppose it's possible that you might get different advice since
you're doing RAID 1 and I was doing RAID 5.  I'm not an expert.  I
just wanted to warn you that I ran into trouble basically doing what
you propose.  The advice I got was that I shouldn't grow and shrink to
replace a disk.  There's a --replace option for that.  That'll remove
a few of your steps, simplify and speed up the process, and remove
risk.  Hope this helps.  Hopefully someone who knows a lot more than I
do will also speak up.  Good luck.

              Eddie

Re: Replacing a RAID1 drive that has not failed.

[Doug Herr]

Thanks to both Wol and Eddie for pointing me to the --replace option and 
for not simply telling me to RTFM.  Part of my confusion was based on my 
last attempt (years back) to find out if three drive RAID 1 was even an 
option.  Just as before I found stuff that laughed at the mere idea of a 
three drive mirror.  Seems people were getting too tied up on the word 
"mirror", which is not really a proper name for it in my view.  The 
mirror copy of myself when looking into a mirror is *not* the same in me 
in so many ways.  Oh, and now I think I understand that hardware RAID 
rarely or never allows it but Linux md RAID has it now and maybe had it 
the whole time.

That said, I really should have slogged thru the man page better.  It 
does seem that --replace includes the key feature that I want:

"the  device remains  in  service  during the recovery process to 
increase resilience against multiple failures."


So, just in case somebody wants to see if they can find anything I 
missed, here is my current plan...


 1. Partition new drive (plugged in via external SATA dock)
#fdisk /dev/sdc
(make it an exact match of sda/sdb unless it turns out to be smaller,
 in which case I can shrink /boot to make room.)

 2. Replace sda partition with sdc partition for each RAID.
(Could do all at once but feels safer to do one at a time.)
#mdadm /dev/md123 --replace /dev/sda7 --with /dev/sdc7
#watch /proc/mdstat
#mdadm /dev/md125 --replace /dev/sda6 --with /dev/sdc6
#watch /proc/mdstat
#mdadm /dev/md126 --replace /dev/sda3 --with /dev/sdc3
#watch /proc/mdstat
#mdadm /dev/md1 --replace /dev/sda5 --with /dev/sdc5
#watch /proc/mdstat
#mdadm /dev/md4 --replace /dev/sda2 --with /dev/sdc2
#watch /proc/mdstat

 3. All RAIDs should be "active" and in "[UU]" status, with no sda partitions.
#cat /proc/mdstat

(Consider if this should only be done later/never so that sda can 
 be a form of backup in case of issues during the post replace reboot.)
 4. Prevent the old partitions from ever being pulled into the RAIDs.
#mdadm --zero-superblock /dev/sda7
#mdadm --zero-superblock /dev/sda6
#mdadm --zero-superblock /dev/sda3
#mdadm --zero-superblock /dev/sda5
#mdadm --zero-superblock /dev/sda2

 5. Make a new /boot on sdc.
(using rsync since the language is taken from my daily cron and this is a 
 tad quicker if I end up running it more than once.)
(fstab line needing update: UUID=52b14d98-b284-41a0-a36f-459ae3ae12a7 /boot ext4 defaults 1 2)
#mkfs.ext4 /dev/sdc1
#mkdir /bootnew
#mount /dev/sdc1 /bootnew
#rsync -a --delete /boot/ /bootnew
#grub-install /dev/sdc
#blkid /dev/sdc1
#vim /etc/fstab
#umount /bootnew ; rmdir /bootnew

 6. Power down and swap sdc into sda "slot".

 7. Make sure that all RAIDs are "active" and in "[UU]" status:
#cat /proc/mdstat




-- 
Doug Herr 

Re: Replacing a RAID1 drive that has not failed.

[Wols Lists]

On 27/10/15 16:45, Doug Herr wrote:
>  1. Partition new drive (plugged in via external SATA dock)
> #fdisk /dev/sdc
> (make it an exact match of sda/sdb unless it turns out to be smaller,
>  in which case I can shrink /boot to make room.)

man gdisk.

Look at the "u" option in the "expert" menu - it'll probably save you
some work :-)

Cheers,
Wol

Re: Replacing a RAID1 drive that has not failed.

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 729 bytes --]

On Tue, 27 Oct 2015 21:31:01 +0000
Wols Lists <antlists@youngman.org.uk> wrote:

> On 27/10/15 16:45, Doug Herr wrote:
> >  1. Partition new drive (plugged in via external SATA dock)
> > #fdisk /dev/sdc
> > (make it an exact match of sda/sdb unless it turns out to be smaller,
> >  in which case I can shrink /boot to make room.)
> 
> man gdisk.
> 
> Look at the "u" option in the "expert" menu - it'll probably save you
> some work :-)

Unfamiliar with 'gdisk', but when I need the exact same set of partitions on
another drive, I use sfdisk:

# sfdisk -d /dev/sda > sda.sf
# sfdisk /dev/sdb < sda.sf

You could even do it in one step:

# sfdisk -d /dev/sda | sfdisk /dev/sdb

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Replacing a RAID1 drive that has not failed.

[Doug Herr]

On Wed, 28 Oct 2015 10:28:17 +0500, Roman Mamedov wrote:

> On Tue, 27 Oct 2015 21:31:01 +0000
> Wols Lists <antlists@youngman.org.uk> wrote:
> 
>> On 27/10/15 16:45, Doug Herr wrote:
>> >  1. Partition new drive (plugged in via external SATA dock)
>> > #fdisk /dev/sdc
>> > (make it an exact match of sda/sdb unless it turns out to be smaller,
>> >  in which case I can shrink /boot to make room.)
>> 
>> man gdisk.
>> 
>> Look at the "u" option in the "expert" menu - it'll probably save you
>> some work :-)
> 
> Unfamiliar with 'gdisk', but when I need the exact same set of 
partitions on
> another drive, I use sfdisk:
> 
> # sfdisk -d /dev/sda > sda.sf
> # sfdisk /dev/sdb < sda.sf
> 
> You could even do it in one step:
> 
> # sfdisk -d /dev/sda | sfdisk /dev/sdb


Thanks much to all who have replied and all who post here.

Yup, the sfdisk copy is one that I am ready to use if needed but your 
replies have triggered better googling and I am now planning to use fdisk 
(I am in Fedora 22) to manually partition the new disk.  I don't think I 
have any need for GPT yet but what I realized is that my current set of 
disks was created when fdisk was still starting at sector 63 where now it 
uses 2048.  I don't think it matters much for my current set of Hitachi 
HDT721010SLA360 drives.  It might not even matter for the slightly newer 
"E7K1000" that is on the way to me, but I do want to stick with the 
current standards as best I can.  I have room to tweak it since I have a 
non-RAID /boot partition that is easily large enough to lend space as 
needed.

I do plan to play with GPT to learn about it and to be more ready for the 
next computer if it is the default at that point.

Oh, I also added a step in my plan.  Adding my new partitions as spares 
first will alert me of size errors (if smaller) *before* any of the 
replacement jobs kick off:

For each:
# mdadm /dev/md4 --add-spare /dev/sdc2

If the above does not complain then move forward for each:
# mdadm /dev/md3 --replace /dev/sda2 --with /dev/sdc2

And fear not, I will be updating my backups before moving forward with 
this.

-- 
Doug Herr