Is (non-failed) disk replace possible without loosing redundancy ?

Is (non-failed) disk replace possible without loosing redundancy ?

[Mihail Daskalov]

Hi all,
I am newbie to this list, so forgive me if this was already discussed, and answered. I have searched the archives, but could not find an answer to my question (except something called hot-replace, that I couldn't understand much about).

I would like to replace a disk in a mirror (raid1 array) with a third disk, that is present in the system (it is actually added as a spare to the array).
This is part of some storage reorganization and drive replacement process (change old disks with new ones).
Is there a way to gracefully replace a non-failed drive in an array, without loosing redundancy. 
I think I would like to achieve the following:
1.	Add a third disk to a mirror array
2.	Ask the md subsystem to create triple mirror, and sync information to the third disk.
3.	Remove one of the other 2 disks on the system
4.	The system will stay with 2 up-to-date mirrors of the blocks

This could be called a 'Copy-and-replace' operation. It could have some optimization - for example - maintain only 2 copies of the information (for example blocks that were not yet copied to the third disk will be on disks {1,2}, and blocks that are already copied would be on {1,3} in the case when disk 2 is being replaced by disk 3).

So my questions is how can I achieve this, or similar functionality, and in what version of the kernel and tools, if at all possible?

I specifically would like to underline that I know I can do:
1. Add third disk as spare
2. fail disk 2
3. wait for the resync to finish

I would like to avoid this, as while resync-ing I would only have one copy of the blocks.

Thanks in advance,
Mihail Daskalov

Re: Is (non-failed) disk replace possible without loosing redundancy ?

[Adam Goryachev]

On 12/04/13 01:07, Mihail Daskalov wrote:
> Hi all,
> I am newbie to this list, so forgive me if this was already discussed, and answered. I have searched the archives, but could not find an answer to my question (except something called hot-replace, that I couldn't understand much about).
>
> I would like to replace a disk in a mirror (raid1 array) with a third disk, that is present in the system (it is actually added as a spare to the array).
> This is part of some storage reorganization and drive replacement process (change old disks with new ones).
> Is there a way to gracefully replace a non-failed drive in an array, without loosing redundancy. 
> I think I would like to achieve the following:
> 1.	Add a third disk to a mirror array
> 2.	Ask the md subsystem to create triple mirror, and sync information to the third disk.
> 3.	Remove one of the other 2 disks on the system
> 4.	The system will stay with 2 up-to-date mirrors of the blocks
>
> This could be called a 'Copy-and-replace' operation. It could have some optimization - for example - maintain only 2 copies of the information (for example blocks that were not yet copied to the third disk will be on disks {1,2}, and blocks that are already copied would be on {1,3} in the case when disk 2 is being replaced by disk 3).
>
> So my questions is how can I achieve this, or similar functionality, and in what version of the kernel and tools, if at all possible?
>
> I specifically would like to underline that I know I can do:
> 1. Add third disk as spare
> 2. fail disk 2
> 3. wait for the resync to finish
>
> I would like to avoid this, as while resync-ing I would only have one copy of the blocks.
>
I'm pretty sure this will work:
1) mdadm --manage /dev/md0 --add /dev/sdc1 # Add spare device
2) mdadm  --grow /dev/md0 --level=1 --raid-devices=3 # Grow the array to
three drive raid1
3) mdadm --manage /dev/md0 --fail /dev/sdb1 # After the resync is
finished, fail the old disk2
4) mdadm --manage /dev/md0 --remove /dev/sdb1 # Now remove the old disk2
5) # Now physically remove disk2.
Repeat steps 1, 3, 4, 5 to replace disk1 with disk4.

6) mdadm --grow /dev/md0 --level=1 --raid-devices=2 # Reduce the array
back to a two drive raid1

If you have a newer kernel and/or mdadm (I think very new) then you can
use the replace functionality you have read about.

BTW, you should test the above, or wait for confirmation from someone
else, I haven't actually tried that, and I don't even know if the
command options are exactly right, I'm just running from memory (see man
mdadm) hoping a quick reply will help you out.

Regards,
Adam

-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

Re: Is (non-failed) disk replace possible without loosing redundancy ?

[Keith Keller]

On 2013-04-11, Adam Goryachev <mailinglists@websitemanagers.com.au> wrote:
>
> I'm pretty sure this will work:
> 1) mdadm --manage /dev/md0 --add /dev/sdc1 # Add spare device
> 2) mdadm  --grow /dev/md0 --level=1 --raid-devices=3 # Grow the array to
> three drive raid1
> 3) mdadm --manage /dev/md0 --fail /dev/sdb1 # After the resync is
> finished, fail the old disk2
> 4) mdadm --manage /dev/md0 --remove /dev/sdb1 # Now remove the old disk2
> 5) # Now physically remove disk2.
> Repeat steps 1, 3, 4, 5 to replace disk1 with disk4.
>
> 6) mdadm --grow /dev/md0 --level=1 --raid-devices=2 # Reduce the array
> back to a two drive raid1

This seems like a reasonable way to replace a RAID1 device without
losing redundancy.  Is there any way to accomplish the same sort of
thing in a RAID5 or 6?  I am imagining something like, if I wanted to
replace device 0, I'd create a RAID1 with that device and the new
device, wait for that build to complete, then break the mirror and
remove the old device 0, so that the new drive becomes the new device 0.
I suppose one could create a RAID5 out of RAID1 components where each
RAID1 is initially degraded, but that seems inelegant.  (I think this
option may have been suggested on this list at one point.)

--keith

-- 
kkeller@wombat.san-francisco.ca.us

Re: Is (non-failed) disk replace possible without loosing redundancy ?

[Sam Bingner]

On Apr 11, 2013, at 11:52 AM, Keith Keller <kkeller@wombat.san-francisco.ca.us> wrote:

> On 2013-04-11, Adam Goryachev <mailinglists@websitemanagers.com.au> wrote:
>> 
>> I'm pretty sure this will work:
>> 1) mdadm --manage /dev/md0 --add /dev/sdc1 # Add spare device
>> 2) mdadm  --grow /dev/md0 --level=1 --raid-devices=3 # Grow the array to
>> three drive raid1
>> 3) mdadm --manage /dev/md0 --fail /dev/sdb1 # After the resync is
>> finished, fail the old disk2
>> 4) mdadm --manage /dev/md0 --remove /dev/sdb1 # Now remove the old disk2
>> 5) # Now physically remove disk2.
>> Repeat steps 1, 3, 4, 5 to replace disk1 with disk4.
>> 
>> 6) mdadm --grow /dev/md0 --level=1 --raid-devices=2 # Reduce the array
>> back to a two drive raid1
> 
> This seems like a reasonable way to replace a RAID1 device without
> losing redundancy.  Is there any way to accomplish the same sort of
> thing in a RAID5 or 6?  I am imagining something like, if I wanted to
> replace device 0, I'd create a RAID1 with that device and the new
> device, wait for that build to complete, then break the mirror and
> remove the old device 0, so that the new drive becomes the new device 0.
> I suppose one could create a RAID5 out of RAID1 components where each
> RAID1 is initially degraded, but that seems inelegant.  (I think this
> option may have been suggested on this list at one point.)
> 
> --keith
> 
> -- 
> kkeller@wombat.san-francisco.ca.us
> 

For a raid5 it's easy -- convert to a raid6 (and let it complete with an extra disk) for the conversion...  then fail, remove, and add disks one at a time while retaining a raid5 level of redundancy.  For a raid6 I don't think you have any such options.

For a raid0 - convert to a raid4, add a parity disk and let it sync, then replace the non-parity disk and resync, last convert back to raid0

Of course all these options require you to have room for an extra disk ;)

Sam

Re: Is (non-failed) disk replace possible without loosing redundancy ?

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 426 bytes --]

On Thu, 11 Apr 2013 14:52:59 -0700
Keith Keller <kkeller@wombat.san-francisco.ca.us> wrote:

> Is there any way to accomplish the same sort of
> thing in a RAID5 or 6?

There is the "want_replacement" mechanism. It exists, it works perfectly, but
from this thread it looks like no one knows about it. Actually I am not sure
where it is documented, try searching the mail list archives.


-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Is (non-failed) disk replace possible without loosing redundancy ?

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 663 bytes --]

On Fri, 12 Apr 2013 16:01:24 +0600 Roman Mamedov <rm@romanrm.ru> wrote:

> On Thu, 11 Apr 2013 14:52:59 -0700
> Keith Keller <kkeller@wombat.san-francisco.ca.us> wrote:
> 
> > Is there any way to accomplish the same sort of
> > thing in a RAID5 or 6?
> 
> There is the "want_replacement" mechanism. It exists, it works perfectly, but
> from this thread it looks like no one knows about it. Actually I am not sure
> where it is documented, try searching the mail list archives.
> 
> 

Checkout the devel mdadm from git://neil.brown.name/mdadm/

 make man
 less mdadm.man

and search for "--replace"

Requires linux-3.3 or later.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Is (non-failed) disk replace possible without loosing redundancy ?

[Keith Keller]

On 2013-04-12, NeilBrown <neilb@suse.de> wrote:
>
> On Fri, 12 Apr 2013 16:01:24 +0600 Roman Mamedov <rm@romanrm.ru> wrote:
>
>> On Thu, 11 Apr 2013 14:52:59 -0700
>> Keith Keller <kkeller@wombat.san-francisco.ca.us> wrote:
>> > Is there any way to accomplish the same sort of
>> > thing in a RAID5 or 6?
>> There is the "want_replacement" mechanism. It exists, it works perfectly,=
>  but
>> from this thread it looks like no one knows about it. Actually I am not s=
> ure
>> where it is documented, try searching the mail list archives.

Actually, I had seen want_replacement before, but had only skimmed
messages about it.  I apologize for not realizing what it was--I should
have remembered that feature and followed up on it first.

> Checkout the devel mdadm from git://neil.brown.name/mdadm/
>
>  make man
>  less mdadm.man
>
> and search for "--replace"
>
> Requires linux-3.3 or later.

This is probably one reason why want_replacement didn't stick in my
head; I'm not quite ready for 3.3.  But perhaps this will encourage me
to try to move up more quickly.

Thanks all!

--keith

-- 
kkeller@wombat.san-francisco.ca.us