*biiiiig* thanks for all developers of bitmap-based raid5 resyncing and
bad block rewriting, both of them is really great feature! :)


 I ported my "proactive thing" to the new kernel, it can live nicely
with the new features now.. some bugs hunted in the last month, it's
quite stable for me

 if you wanna give a try you must apply Neil's badblock rewriting patch
first, attached also



- *readme* -

 This is a feature patch that implements 'proactive raid5 disk
replacement' (http://www.arctic.org/~dean/raid-wishlist.html),
that could help a lot on large raid5 arrays built from cheap sata
drivers when the IO traffic such large that daily media scan on the
disks isn't possible.
 An atypical breakdown situation is when a drive gets kicked from the
array due to a bad block, I replace it but the resync fails cause
another 2-3 disks has hidden badblocks too. In this situation I've
to save the disks with dd and rebuild bad blocks with a userspace
tool (by hand), meanwhile the site is down for hours. This patch
tries to give a solution for this problem, the two main feature is:
 1. Don't kick a drive on read error cause it is possible that 99.99% is
useable and will help (to serve and to save data) if another drive shows
bad sectors in same array - Neil's new (experimental) sector rewrite
feature included, the first thing is always a try to rewrite the bad
sector
 2. Let to mirror a partially failed drive to a spare _online_ and
replace the source of the mirror with the spare when it's done. Bad
blocks isn't a problem unless same stripe is damaged on two disks what's
a rare case. In this way is possible to fix an array with partially
failed drives without data loss and without downtime. In other words,
you never have to degrade the array due to a disk change, you can do
that in optimal state.

 Per-device bad block cache is implemented to speed up arrays with
partially failed drives (replies are often slow from those), also
helps to determine badly damaged drives based on number of bad blocks,
and can take an action if steps over an user defined threshold
(see /proc/sys/dev/raid/badblock_tolerance). Rewrite of a bad block
will delete the entry from the cache.
 performance is affected just a little bit if there's no or some
registered bad blocks, but over a million that could be a problem
currently..

 Some words about error handling: first big change is now you can use an
external error handler, what means that a user-space script will be
called by the kernel to handle the situation. The common method in this
script is to call 'mdadm' and choosing return values (see below). This
is good -for example- if you have 1 spare drive for 2 arrays, a script
can handle it nicely.. If the script has failed to run (or does not
exists), there's a default algorithm, the main guidelines of that:
 - a "disk fail" means that it's oversteps the 'badblock threshold' or
    failed on write
 - if a drive fails in an optimal array and there's no spare the
    disk will be kicked from the array
 - if the drive fails in degraded array the drive _won't_ be kicked.
    processes gets read/write error if data is needed from the
    damaged sectors. if you want the old behavior use an external
    error handler
 - if drive fails and there's a spare then the proactive mirroring
    begins to the spare. the failing drive won't be kicked until
    the mirror has not been done


 Well, better if you know, It's an ugly hack, I'm not a kernel guru,
but I love the idea and now I can't live without it on my own servers
(so this is works for me). I hope somebody will implement this
feature once in a much nicer adaptation, I'm trying to maintain this
patch till then..


You should put your external error handler script at location
"/sbin/mdevent"; it gets the following arguments:
 1st: name of the md array (eg.: "md0")
 2nd: kind of the fail event as string, currently always "drivefail"
 3rd: name of the drive (maybe major/minor nr would be better, currently
			 you can translate to that by /proc/partitions)

Let's see how can you handle some situations from the script:
  array is optimal, a disk fails:
    you want to.. fail that drive and add a spare for normal rebuilding
	mdadm -f /dev/$2 /dev/$3
	mdadm -a /dev/$2 /dev/my_spare1
	exit 0
    ..start proactive mirroring of that disk
	mdadm -a /dev/$2 /dev/my_spare1
	exit 0
    ..keep it on and reset the badblock cache
	exit 1
    ..just keep it in sync
	exit 0
    ..let the default action
	exit 2
    Notice that if the proactive mirroring is done the spare won't
    replace the source drive automatically, you should do it by hand
    or by a scheluded task. You've got a last chance to re-think it.


 (raid6 could be another solution for this problem, but that's the
 big far evil in my eyes ;)


use:

1. patch the kernel, this one is against 2.6.14
2. type:

# make the drives
mdadm -B -n1 -l faulty -c4 /dev/md/1 /dev/rd/0
mdadm -B -n1 -l faulty -c4 /dev/md/2 /dev/rd/1
mdadm -B -n1 -l faulty -c4 /dev/md/3 /dev/rd/2

# make the array
mdadm -C -n3 -l5 /dev/md/0 /dev/md/1 /dev/md/2 /dev/md/3

# .. wait for sync ..

# grow bad blocks as ma*tor does :)
mdadm --grow -l faulty -p rp454 /dev/md/1
mdadm --grow -l faulty -p rp738 /dev/md/2

# add a spare
mdadm -a /dev/md/0 /dev/rd/4

# -> fail a drive, sync begins <-
#  the md/1 will not be marked as failed, this is the point, but
#  if you want to, you can issue this command again
mdadm -f /dev/md/0 /dev/md/1

# kernel:
#  resync from md1 to spare ram4
#  added spare for active resync

# .. wonder the read errors from md[12] and the sync goes on!
# feel free to stress the md at this time, mkfs, dd, badblocks, etc

# kernel:
#  raid5_spare_active: 3 in_sync 3->0
# /proc/mdstat:
#  md0 : active raid5 ram4[0] md3[2] md2[1] md1[0]
# -> ram4 and md1 has same id, this means the spare is a complete mirror,
#       if you stop the array you can assembly it with ram4 instead of md1,
#       the superblock is same on them

# check the mirror (stop write stress if any)
mdadm --grow -l faulty -p none /dev/md/1
cmp /dev/md/1 /dev/rd/4

# hot-replace the mirrored -partially failed- device with the active spare
#  (yes, mark it as failed again, but if there's a syncing- or synced 'active spare'
#       the -f really fails the device or replace it with the synced spare)
mdadm -f /dev/md/0 /dev/md/1

# kernel:
#  replace md1 with in_sync active spare ram4

# and voila!
# /proc/mdstat:
#  md0 : active raid5 ram4[0] md3[2] md2[1]



--
 dap