Re: [md PATCH 00/16] hot-replace support for RAID4/5/6

[NeilBrown <neilb@suse.de>]

[-- Attachment #1: Type: text/plain, Size: 6667 bytes --]

On Thu, 27 Oct 2011 11:10:34 -0600 "Peter W. Morreale" <morreale@sgi.com>
wrote:

> On Wed, 2011-10-26 at 12:43 +1100, NeilBrown wrote: 
> > The following series - on top of my for-linus branch which should appear in
> > 3.2-rc1 eventually - implements hot-replace for RAID4/5/6.  This is almost
> > certainly the most requested feature over the last few years.
> > The whole series can be pulled from my md-devel branch:
> >    git://neil.brown.name/md md-devel
> > (please don't do a full clone, it is not a very fast link).
> > 
> > There is currently no mdadm support, but you can test it out and
> > experiment without mdadm.
> > 
> > In order to activate hot-replace you need to mark the device as
> > 'replaceable'.
> > This happens automatically when a write error is recorded in a
> > bad-block log (if you happen to have one).
> > It can be achieved manually by
> >    echo replaceable > /sys/block/mdXX/md/dev-YYY/state
> > 
> > This makes YYY, in XX, replaceable.
> > 
> > If md notices that there is a replaceable drive and a spare it will
> > attach the spare to the replaceable drive and mark it as a
> > 'replacement'.
> > This word appears in the 'state' file and as (R) in /proc/mdstat.
> > 
> > md will then copy data from the replaceable drive to the replacement.
> > If there is a bad block on the replaceable drive, it will get the data
> > from elsewhere.  This looks like a "recovery" operation.
> > 
> > When the replacement completes the replaceable device will be marked
> > as Failed and will be disconnected from the array (i.e. the 'slot'
> > will be set to 'none') and the replacement drive will take up full
> > possession of that slot.
> 
> Neil,
> 
> Seems to work quite well.  Note I have not yet performed a data
> consistency check, just the mechanics of 'replacing' an existing
> drive.  
> 
> I see in the code that a recovery is kicked immediately after changing
> the state of a drive.  One question is whether it will be possible to
> mark multiple drives for replacement, then invoke the recovery one time,
> replacing all disks marked in a single pass?
> 
> Right now, it changing state on multiple drives kicks off sequential
> recoveries.  For larger disks (3TB/etc), recovery takes a long time and
> there is a non-zero performance hit on the live array.
> 
> There are two common use cases to think about.  First being an array
> disk replacement to (say) larger disks.  Second being a new array in use
> for a period of time where the disks are approaching end-of-life, and
> multiple disks are showing signs of possible failure.  So we want to
> replace a number of them at one time and incur the performance hit one
> time. 
> 
> I see where the code limits a recovery to one sync at a time, would it
> be possible to extend this to multiple concurrent replacements?
> 
> What would it take to enable this?

echo frozen > /sys/block/mdX/md/sync_action
for i in /sys/block/mdX/md/dev-*/state
do echo replaceable > $i
done
echo repair > /sys/block/mdX/md/sync_action

should do it.  You certainly should be able to replace several devices at the
same time using this approach, though I haven't tried it.

(hmmm... it probably shouldn't accept a 'replaceable' flag on spares - I'll
make a note of that).

> 
> Thanks again for this effort, this is terrific. 

Thanks.

NeilBrown


> 
> Best,
> -PWM
> 
> 
> > 
> > It is not possible to assemble an array with replacement with mdadm.
> > To do this by hand:
> > 
> >   mknod /dev/md27 b 9 27
> >   < /dev/md27
> >   cd /sys/block/md27/md
> >   echo 1.2 > metadata_version
> >   echo 8:1 > new_dev
> >   echo 8:17 > new_dev
> >    ...
> >   echo active > array_state
> > 
> > Replace '27' by the md number you want.  Replace 1.2 by the metadata
> > version number (must be 1.x for some x).  Replace 8:1, 8:17 etc
> > by the major:minor numbers of each device in the array.
> > 
> > Yes: this is clumsy.  But they you aren't doing this on live data -
> > only on test devices to experiment.
> > 
> > You can still assemble the array without the replacement using mdadm.
> > Just list all the drives except the replacement in the --assemble
> > command.
> > Also once the replacement operation completes you can of course stop
> > and assemble the new array with old mdadm.
> > 
> > I hope to submit this together with support for RAID10 (and maybe some
> > minimal support for RAID1) for Linux-3.3. By the time it comes out
> > mdadm-3.3 should exist will full support for hot-replace.
> > 
> > Review and testing is very welcome, be please do not try it on live
> > data.
> > 
> > NeilBrown
> > 
> > 
> > ---
> > 
> > NeilBrown (16):
> >       md/raid5: Mark device replaceable when we see a write error.
> >       md/raid5: If there is a spare and a replaceable device, start replacement.
> >       md/raid5: recognise replacements when assembling array.
> >       md/raid5: handle activation of replacement device when recovery completes.
> >       md/raid5:  detect and handle replacements during recovery.
> >       md/raid5: writes should get directed to replacement as well as original.
> >       md/raid5: allow removal for failed replacement devices.
> >       md/raid5: preferentially read from replacement device if possible.
> >       md/raid5: remove redundant bio initialisations.
> >       md/raid5: raid5.h cleanup
> >       md/raid5: allow each slot to have an extra replacement device
> >       md: create externally visible flags for supporting hot-replace.
> >       md: change hot_remove_disk to take an rdev rather than a number.
> >       md: remove test for duplicate device when setting slot number.
> >       md: take after reference to mddev during sysfs access.
> >       md: refine interpretation of "hold_active == UNTIL_IOCTL".
> > 
> > 
> >  Documentation/md.txt      |   22 ++
> >  drivers/md/md.c           |  132 ++++++++++---
> >  drivers/md/md.h           |   82 +++++---
> >  drivers/md/multipath.c    |    7 -
> >  drivers/md/raid1.c        |    7 -
> >  drivers/md/raid10.c       |    7 -
> >  drivers/md/raid5.c        |  462 +++++++++++++++++++++++++++++++++++----------
> >  drivers/md/raid5.h        |   98 +++++-----
> >  include/linux/raid/md_p.h |    7 -
> >  9 files changed, 599 insertions(+), 225 deletions(-)
> > 
> > -- 
> > Signature
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]