Safely swapping a disk in a RAID456

Safely swapping a disk in a RAID456

[martin f krafft]

[-- Attachment #1: Type: text/plain, Size: 1012 bytes --]

Dear list,

I would like to swap out a drive in a RAID6 (4 drives), but ideally
without letting the RAID degrade. I was thinking that it should be
possible to declare the new disk a copy of the old one, let it sync,
then remove the old one and let the new one take over, but MD does
not seem to support that.

Next, I thought that I could add the new disk as a spare, grow the
array to 5 disks, and somehow shrink it back to 4 again, but of
course that does not work either, since 4→5 is a reshape, and there
seems to be no way to control which disk to remove from the RAID6 on
a shrink (making it a spare), without degrading the array in the
process.

Does anyone have any ideas? This should not be so hard…

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"courage is not the absence of fear, but the decision
 that something else is more important than fear."
                                                  -- ambrose redmoon
 
spamtraps: madduck.bogus@madduck.net

[-- Attachment #2: Digital signature (see http://martin-krafft.net/gpg/sig-policy/999bbcc4/current) --]
[-- Type: application/pgp-signature, Size: 1124 bytes --]

Re: Safely swapping a disk in a RAID456

[Rudy Zijlstra]

Hi,

On 09/30/2011 05:42 PM, martin f krafft wrote:
> Dear list,
>
> I would like to swap out a drive in a RAID6 (4 drives), but ideally
> without letting the RAID degrade. I was thinking that it should be
> possible to declare the new disk a copy of the old one, let it sync,
> then remove the old one and let the new one take over, but MD does
> not seem to support that.
If you can stop the array it can be done. Otherwise you will have to 
degrade it while replacing

Replacing without degrade:
- stop the raid
- do a dd from old disk to new disk
- replace the disk
- re-start the raid

Cheers,


Rudy

Re: Safely swapping a disk in a RAID456

[Andre Noll]

[-- Attachment #1: Type: text/plain, Size: 517 bytes --]

On Fri, Sep 30, 18:06, Rudy Zijlstra wrote:
> Replacing without degrade:
> - stop the raid
> - do a dd from old disk to new disk
> - replace the disk
> - re-start the raid

This works but the array needs to be stopped during the dd, which may take a
long time. Creating a raid1 first allows you to restart the raid456 immediately
after the raid1 has been started, no need to wait for the initialization to
complete.

Andre
-- 
The only person who always got his work done by Friday was Robinson Crusoe

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Safely swapping a disk in a RAID456

[Andre Noll]

[-- Attachment #1: Type: text/plain, Size: 358 bytes --]

On Fri, Sep 30, 17:42, martin f krafft wrote:
> Does anyone have any ideas?

Create a 2-disk Raid1 (with external metadata) containing the disk
you'd like to replace and the replacement disk.

And yes, it would be great if md provided an easier way to do this.

Andre
-- 
The only person who always got his work done by Friday was Robinson Crusoe

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Safely swapping a disk in a RAID456

[David Brown]

On 30/09/11 18:28, Andre Noll wrote:
> On Fri, Sep 30, 17:42, martin f krafft wrote:
>> Does anyone have any ideas?
>
> Create a 2-disk Raid1 (with external metadata) containing the disk
> you'd like to replace and the replacement disk.
>
> And yes, it would be great if md provided an easier way to do this.
>

That would be "hot replace", which is quite high up on the md/mdadm 
"things to do" list.

<http://neil.brown.name/blog/20110216044002#2>

Re: Safely swapping a disk in a RAID456

[Andre Noll]

[-- Attachment #1: Type: text/plain, Size: 1235 bytes --]

On Sat, Oct 01, 16:53, David Brown wrote:
> On 30/09/11 18:28, Andre Noll wrote:
> >On Fri, Sep 30, 17:42, martin f krafft wrote:
> >>Does anyone have any ideas?
> >
> >Create a 2-disk Raid1 (with external metadata) containing the disk
> >you'd like to replace and the replacement disk.
> >
> >And yes, it would be great if md provided an easier way to do this.
> >
> 
> That would be "hot replace", which is quite high up on the md/mdadm
> "things to do" list.
> 
> <http://neil.brown.name/blog/20110216044002#2>

Thanks for the pointer. I wonder if it would be useful to have an
md-aware dd command which could read from an md device and output
only the contents of one component. I.e. it would give the same result
as dd if=/dev/component-device of=/dev/new-disk if there are no read
errors but would redirect failed reads to the working devices.

This should be easy to implement as it avoids the complexity mentioned
in Neil's blog posting. Moreover, it could be written purely in user
space and would work on any kernel version.  The price to pay is of
course that the array needs to be r/o during the operation.

Andre
-- 
The only person who always got his work done by Friday was Robinson Crusoe

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Safely swapping a disk in a RAID456

[John Robinson]

On 30/09/2011 16:42, martin f krafft wrote:
> Dear list,
>
> I would like to swap out a drive in a RAID6 (4 drives), but ideally
> without letting the RAID degrade. I was thinking that it should be
> possible to declare the new disc a copy of the old one, let it sync,
> then remove the old one and let the new one take over, but MD does
> not seem to support that.

Not yet, but it's in the roadmap as a hot-replace. In the mean time, you 
can do something very similar by hand with almost no lost of redundancy, 
if your RAID456 has a bitmap.

> Next, I thought that I could add the new disc as a spare, grow the
> array to 5 discs, and somehow shrink it back to 4 again, but of
> course that does not work either, since 4→5 is a reshape, and there
> seems to be no way to control which disc to remove from the RAID6 on
> a shrink (making it a spare), without degrading the array in the
> process.

Doing a shrink while specifying which drive to remove is also on the 
roadmap, I believe. But doing huge reshapes like this are not the way 
forward for a hot-replace.

> Does anyone have any ideas? This should not be so hard…

I'd be glad to hear of any "real" RAID card that made it easier, or even 
possible.

Something along the lines of:

# briefly remove your disc from the array
mdadm --manage /dev/md_raid456 --remove /dev/the_disc_i_want_to_remove
# make a temp raid1 with only the one disc, using build so metadata is
# only in RAM and nothing is written to the disc
mdadm --build /dev/md_temp_raid1 --level 1 
/dev/the_disc_i_want_to_remove missing
# now re-add something which looks identical to your original disc,
# but is actually a single-sided mirror, back into the array
# the above can all be done very quickly so your raid456 only runs
# without a drive for seconds
mdadm --manage /dev/md_raid456 --re-add /dev/md_temp_raid1
# now get md to make a mirror (copy) to the new disc
mdadm --manage /dev/md_temp_raid1 --add /dev/the_new_disc
# wait for it to finish, or just wait for it yourself
mdadm --wait /dev/md_temp_raid1
# and switch back again: remove the temporary raid1
mdadm --manage /dev/md_raid456 --remove /dev/md_temp_raid1
# stop it so the new disc becomes available again
mdadm --stop /dev/md_temp_raid1
# and put the new disc which is now a complete copy of the old one
# back in to the array
mdadm --manage /dev/md_raid456 --re-add /dev/the_new_disc
# and you're done

This is all fine if your old disc has no faulty sectors. If it does 
have, you need more help from someone much more clued-up than me 
(because it is already possible to do partial rebuilds by manipulating 
/sys), or mdadm's roadmap feature of hot-replace, which will do the 
above and also automatically perform partial rebuilds from the rest of 
the array when the old disc has bad sectors.

Hope this helps.

Cheers,

John.

-- 
John Robinson, yuiop IT services
0131 557 9577 / 07771 784 058
46/12 Broughton Road, Edinburgh EH7 4EE
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Safely swapping a disk in a RAID456

[Danny Rawlins]

John Robinson wrote:
> On 30/09/2011 16:42, martin f krafft wrote:
>> Dear list,
>>
>> I would like to swap out a drive in a RAID6 (4 drives), but ideally
>> without letting the RAID degrade. I was thinking that it should be
>> possible to declare the new disc a copy of the old one, let it sync,
>> then remove the old one and let the new one take over, but MD does
>> not seem to support that.
>
> Not yet, but it's in the roadmap as a hot-replace. In the mean time,
> you can do something very similar by hand with almost no lost of
> redundancy, if your RAID456 has a bitmap.
>
>> Next, I thought that I could add the new disc as a spare, grow the
>> array to 5 discs, and somehow shrink it back to 4 again, but of
>> course that does not work either, since 4→5 is a reshape, and there
>> seems to be no way to control which disc to remove from the RAID6 on
>> a shrink (making it a spare), without degrading the array in the
>> process.
>
> Doing a shrink while specifying which drive to remove is also on the
> roadmap, I believe. But doing huge reshapes like this are not the way
> forward for a hot-replace.
>
>> Does anyone have any ideas? This should not be so hard…
>
> I'd be glad to hear of any "real" RAID card that made it easier, or
> even possible.
>
> Something along the lines of:
>
> # briefly remove your disc from the array
> mdadm --manage /dev/md_raid456 --remove /dev/the_disc_i_want_to_remove
> # make a temp raid1 with only the one disc, using build so metadata is
> # only in RAM and nothing is written to the disc
> mdadm --build /dev/md_temp_raid1 --level 1
> /dev/the_disc_i_want_to_remove missing
> # now re-add something which looks identical to your original disc,
> # but is actually a single-sided mirror, back into the array
> # the above can all be done very quickly so your raid456 only runs
> # without a drive for seconds
> mdadm --manage /dev/md_raid456 --re-add /dev/md_temp_raid1
> # now get md to make a mirror (copy) to the new disc
> mdadm --manage /dev/md_temp_raid1 --add /dev/the_new_disc
> # wait for it to finish, or just wait for it yourself
> mdadm --wait /dev/md_temp_raid1
> # and switch back again: remove the temporary raid1
> mdadm --manage /dev/md_raid456 --remove /dev/md_temp_raid1
> # stop it so the new disc becomes available again
> mdadm --stop /dev/md_temp_raid1
> # and put the new disc which is now a complete copy of the old one
> # back in to the array
> mdadm --manage /dev/md_raid456 --re-add /dev/the_new_disc
> # and you're done
>
> This is all fine if your old disc has no faulty sectors. If it does
> have, you need more help from someone much more clued-up than me
> (because it is already possible to do partial rebuilds by manipulating
> /sys), or mdadm's roadmap feature of hot-replace, which will do the
> above and also automatically perform partial rebuilds from the rest of
> the array when the old disc has bad sectors.
>
> Hope this helps.
>
> Cheers,
>
> John.
>
For bad disks I use GNU ddrescue in two pass like this.

ddrescue -n -f /dev/sda /dev/sdb ddrescue.log
ddrescue -d -f -r 10 /dev/sda /dev/sdb ddrescue.log

I recently found partclone can make a domain-log for ddrescue to only
rescue from used file system blocks (but not applicable to this situation)

I should get in touch with the GNU ddrescue author to see if (s)he is
willing to restructure the ddrescue binary as a library and the ddresuce
front end then tings like lvm2 and mdadm could put the ddrescue library
to use in recovering as much as they can from said faulty disk before
discarding it.

PS great idea on adding a bitmap to the raid6 removing disk making a
raid1with external metadata then syncing both raid1 disks then removing
the raid1 from the raid6 and then breaking the raid1 to put the new disk
back into the raid1 group. Seems a little over kill for raid6 but that
would make more sense for raid5, though taking less chances on raid6
hitting some sleeping bad blocks (which i like to call them landmines)
is a good thing.

Regards,
Danny Rawlins
Romster @ freenode
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Safely swapping a disk in a RAID456

[Asdo]

On 09/30/11 20:02, John Robinson wrote:
>
> Not yet, but it's in the roadmap as a hot-replace. In the mean time, 
> you can do something very similar by hand with almost no lost of 
> redundancy, if your RAID456 has a bitmap.
> ...
>
> I'd be glad to hear of any "real" RAID card that made it easier, or 
> even possible.
>
> Something along the lines of:
>
> # briefly remove your disc from the array
> mdadm --manage /dev/md_raid456 --remove /dev/the_disc_i_want_to_remove
> # make a temp raid1 with only the one disc, using build so metadata is
> # only in RAM and nothing is written to the disc
> mdadm --build /dev/md_temp_raid1 --level 1 
> /dev/the_disc_i_want_to_remove missing
> # now re-add something which looks identical to your original disc,
> # but is actually a single-sided mirror, back into the array
> # the above can all be done very quickly so your raid456 only runs
> # without a drive for seconds
> mdadm --manage /dev/md_raid456 --re-add /dev/md_temp_raid1
> # now get md to make a mirror (copy) to the new disc
> mdadm --manage /dev/md_temp_raid1 --add /dev/the_new_disc
> # wait for it to finish, or just wait for it yourself
> mdadm --wait /dev/md_temp_raid1
> # and switch back again: remove the temporary raid1
> mdadm --manage /dev/md_raid456 --remove /dev/md_temp_raid1
> # stop it so the new disc becomes available again
> mdadm --stop /dev/md_temp_raid1
> # and put the new disc which is now a complete copy of the old one
> # back in to the array
> mdadm --manage /dev/md_raid456 --re-add /dev/the_new_disc
> # and you're done

Very good, but...

> This is all fine if your old disc has no faulty sectors. If it does 
> have, you need more help from someone much more clued-up than me 
> (because it is already possible to do partial rebuilds by manipulating 
> /sys), 

This is the problem. And if you want to replace a drive, it's probably 
because it has bad sectors.

I think that during raid1 device-add (which automatically initiates 
rebuild of raid1), the raid1 would go down completely as soon as it hits 
bad sectors.

So you wouldn't be in a position to use rebuilds by /sys "afterwards", 
because there is no "afterwards".

For this to work, somebody would need to implement the bad block list 
also on the legacy array without metadata (--build), so that the raid1 
would stay up even on bad reads during its rebuild. Actually I don't 
know if Neil implemented that or not, I have not tested bad blocks list 
yet, I am intuitively assuming that he improved only the last version of 
the arrays, that is 1.x .

> or mdadm's roadmap feature of hot-replace, which will do the above and 
> also automatically perform partial rebuilds from the rest of the array 
> when the old disc has bad sectors.

the hot-replace is the most wanted feature as of now.