Automatically adding spares on boot

Automatically adding spares on boot

[Can Jeuleers]

[-- Attachment #1: Type: text/plain, Size: 537 bytes --]

Hi,

Over the years I've had various problems with the degree to which the
adding of spares on boot is automatic. There have been times when this
has just worked, and other times (such as now) where it only works very
occasionally (or not at all). Google is full of stories like this.

I have no clue why it's not working automatically, I just know that it
isn't.

So I wrote a little script that automates this, and hooked it up to
mdmon as an event handler.

It can probably do with an elegance upgrade, but it works for me.

HTH, Jan

[-- Attachment #2: mdmon_eventhandler --]
[-- Type: text/plain, Size: 1455 bytes --]

#!/bin/bash
# mdadm --monitor event handler, by Jan Ceuleers
# Licence: GPL v2
# Handles SparesMissing event. Uses blkid output to find devices that have
# the same UUID as the RAID device, but which are not yet members of the array.
# Have this script executed by mdadm by passing its name to the mdadm --monitor
# daemon using the --program parameter.
# On Debian-like systems this can be accomplished like so:
#   echo "DAEMON_OPTIONS=\"\$DAEMON_OPTIONS --program /pathto/script\"" >> /etc/default/mdadm

BLKID="/sbin/blkid"
MDADM="/sbin/mdadm"

(
  case $1 in
    SparesMissing)
      COMPUUID=`$MDADM  -D $2 | grep UUID | cut -f 2 -d 'D' | cut -f 3 -d ' '`
      COMPUUID2=${COMPUUID//:/}
      COMPUUID3=${COMPUUID2:0:8}-${COMPUUID2:8:4}-${COMPUUID2:12:4}-${COMPUUID2:16:4}-${COMPUUID2:20}
      echo The UUID of component devices is $COMPUUID \($COMPUUID3\)
      blkid | grep $COMPUUID3 | grep linux_raid_member | (
        while read l; do
          COMPDEV=`echo $l | cut -f 1 -d ':'`
          $MDADM -D $2 | grep $COMPDEV > /dev/null
          if [ $? -eq 0 ] ; then
            echo Device $COMPDEV is already a member of the array
          else
            echo Found device $COMPDEV with the same UUID but not yet a component device.
            $MDADM $2 --add $COMPDEV 2>&1
          fi
        done
      )
      ;;
    *)
      echo "Unknown event type; can't handle."
      ;;
  esac

) | mail -s "Handling $1 mdadm event on $2" root

Re: Automatically adding spares on boot

[Can Jeuleers]

On 01/09/2013 06:09 PM, Can Jeuleers wrote:
> Hi,
> 
> Over the years I've had various problems with the degree to which the
> adding of spares on boot is automatic. There have been times when this
> has just worked, and other times (such as now) where it only works very
> occasionally (or not at all). Google is full of stories like this.
> 
> I have no clue why it's not working automatically, I just know that it
> isn't.
> 
> So I wrote a little script that automates this, and hooked it up to
> mdmon as an event handler.
> 
> It can probably do with an elegance upgrade, but it works for me.
> 
> HTH, Jan

Here is sample output of the script, which it mails to root:

The UUID of component devices is 442e9934:97191d8e:6d0cf7a9:41621837
(442e9934-9719-1d8e-6d0c-f7a941621837)
Device /dev/sdb2 is already a member of the array
Device /dev/sdd2 is already a member of the array
Found device /dev/sde3 with the same UUID but not yet a component device.
mdadm: re-added /dev/sde3

It's a great shame that there does not appear to be a standard format
for representing UUIDs: mdadm uses an 8:8:8:8 format, whereas blkid
wants an 8-4-4-4-12 format. Hence the conversion shenanigans in the
script. Makes the code brittle.

Cheers, Jan

Re: Automatically adding spares on boot

[Roy Sigurd Karlsbakk]

> Over the years I've had various problems with the degree to which the
> adding of spares on boot is automatic. There have been times when this
> has just worked, and other times (such as now) where it only works
> very
> occasionally (or not at all). Google is full of stories like this.
> 
> I have no clue why it's not working automatically, I just know that it
> isn't.
> 
> So I wrote a little script that automates this, and hooked it up to
> mdmon as an event handler.
> 
> It can probably do with an elegance upgrade, but it works for me.

hm… I don't se the point, really. I have a RAID-6 with a spare, and last time a drive died, the spare kicked in automatically. Why would you add something at boot?
-- 
Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
roy@karlsbakk.net
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Automatically adding spares on boot

[Can Jeuleers]

On 01/19/2013 07:27 PM, Roy Sigurd Karlsbakk wrote:
>> Over the years I've had various problems with the degree to which the
>> adding of spares on boot is automatic. There have been times when this
>> has just worked, and other times (such as now) where it only works
>> very
>> occasionally (or not at all). Google is full of stories like this.
>>
>> I have no clue why it's not working automatically, I just know that it
>> isn't.
>>
>> So I wrote a little script that automates this, and hooked it up to
>> mdmon as an event handler.
>>
>> It can probably do with an elegance upgrade, but it works for me.
> 
> hm… I don't se the point, really. I have a RAID-6 with a spare, and last time a drive died, the spare kicked in automatically. Why would you add something at boot?

For the spare to be automatically swung into action when a drive fails,
the spare must first have been added to the array. In other words: it
can't just be an unused drive somewhere: mdadm must know that it has
permission to use this drive (or partition or whatever) as a hot spare.

Normally you would expect mdadm's auto-assembly code, or the udev
scripts that ship with the OS, to add all relevant devices to each
array: not only the active devices but also the spares. The problem I'm
facing is that (for reasons unknown) my arrays start up with only the
active members added, and the spare devices remaining unused.

Specifically, my /proc/mdstat looks like this upon boot:

Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5]
[raid4] [raid10]
md0 : active raid1 sdb1[1] sdd1[0]
      521984 blocks [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md1 : active raid1 sdb2[1] sdd2[0]
      487861824 blocks [2/2] [UU]
      bitmap: 1/4 pages [4KB], 65536KB chunk

unused devices: <none>

Note that mdadm doesn't think that there are any unused devices, but
there are. Because when mdadm --monitor notices that each of the above
arrays should have a spare it throws a SparesMissing event, which
triggers the script that I sent to the list. Once both of these events
have been handled /proc/mdstat looks like this:

Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5]
[raid4] [raid10]
md0 : active raid1 sde2[2](S) sdb1[1] sdd1[0]
      521984 blocks [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md1 : active raid1 sde3[2](S) sdb2[1] sdd2[0]
      487861824 blocks [2/2] [UU]
      bitmap: 1/4 pages [4KB], 65536KB chunk

unused devices: <none>

Hope this clarifies. You don't need this script if your OS always finds
your spares and adds them automatically to your arrays. Mine doesn't.

Cheers, Jan
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Automatically adding spares on boot

[Roy Sigurd Karlsbakk]

> Hope this clarifies. You don't need this script if your OS always
> finds
> your spares and adds them automatically to your arrays. Mine doesn't.

Really, if md 'forgets' a spare on reboot, that's a bug and should be reported. This is how my raid looks

Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10] 
md0 : active raid6 sdg[5] sdh[6](S) sdc[7] sdb[0] sde[3] sdf[4] sdd[8]
      7814051840 blocks super 1.2 level 6, 64k chunk, algorithm 2 [6/6] [UUUUUU]
      
unused devices: <none>

What distro/version is this?

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
roy@karlsbakk.net
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Automatically adding spares on boot

[Can Jeuleers]

On 01/20/2013 12:51 PM, Roy Sigurd Karlsbakk wrote:
>> Hope this clarifies. You don't need this script if your OS always
>> finds
>> your spares and adds them automatically to your arrays. Mine doesn't.
> 
> Really, if md 'forgets' a spare on reboot, that's a bug and should be reported.

http://www.google.com/#hl=en&q=mdadm+does+not+add+spares+on+boot

First hit.

root@zotac:~# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 11.10
Release:	11.10
Codename:	oneiric
root@zotac:~# uname -a
Linux zotac 3.0.0-30-generic #47-Ubuntu SMP Wed Jan 2 23:16:29 UTC 2013
x86_64 x86_64 x86_64 GNU/Linux
root@zotac:~# mdadm -V
mdadm - v3.1.4 - 31st August 2010

Re: Automatically adding spares on boot

[Roy Sigurd Karlsbakk]

----- Opprinnelig melding -----
> On 01/20/2013 12:51 PM, Roy Sigurd Karlsbakk wrote:
> >> Hope this clarifies. You don't need this script if your OS always
> >> finds
> >> your spares and adds them automatically to your arrays. Mine
> >> doesn't.
> >
> > Really, if md 'forgets' a spare on reboot, that's a bug and should
> > be reported.
> 
> http://www.google.com/#hl=en&q=mdadm+does+not+add+spares+on+boot
> 
> First hit.

That's rather old…
 
> root@zotac:~# lsb_release -a
> No LSB modules are available.
> Distributor ID: Ubuntu
> Description: Ubuntu 11.10
> Release: 11.10
> Codename: oneiric

That's rather old too, and soon out of support. Better upgrade to 12.04 LTS and try again.

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
roy@karlsbakk.net
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Automatically adding spares on boot

[Can Jeuleers]

On 01/20/2013 01:21 PM, Roy Sigurd Karlsbakk wrote:
> That's rather old too, and soon out of support. Better upgrade to 12.04 LTS and try again.

I do know that.

I reported this bug when the release on which I reported it was still
current. I didn't have this problem on 2 subsequent Ubuntu releases and
now I'm having it again on 11.10. So I did something about it myself and
I posted the solution so others can help themselves too.