RAID-5 spare drive: Can't make spare persist across reboot

RAID-5 spare drive: Can't make spare persist across reboot

[David Rea]

   [First-time poster, done plenty of googling and archive searching
before sending this, so if it's already been covered my apologies in
advance...]

Hello All-

I am running a 3-disk RAID-5 using kernel 2.6.24 and mdadm 2.6.3 under
Ubuntu server 8.04LTS. I'm trying to add a spare drive to the array.

/dev/md0 == /dev/sdc, /dev/sdd, /dev/sda

I want to add /dev/sdb1 as a spare partition, so I tried `mdadm
/dev/md0 --add /dev/sdb1` ... this worked great; running `mdadm
--detail /dev/md0` shows 3 active and 1 spare. Cool!

However, after a reboot, I'm back to 3 active and 0 spare drives. I
tried adding the spare to my DEVICES list in /etc/mdadm/mdadm.conf,
and adding a SPARES=1 attribute to the ARRAY line in that file, still
no persistence. I tried running `update-initramfs -u` after changing
mdadm.conf (per an IRC suggestion) but still no persistence.

-- Is it possible to do what I'm trying to do?
-- Do I need to create the array anew using `mdadm --create` and
specifying "--spare-devices=1 /dev/sdb1" during creation?
-- Will creating the array again risk corrupting the data already present?
-- Is there any way to instantiate a superblock on /dev/sdb1 so that
it can be associated with this array's UUID?
-- I have the ARRAY line (in mdadm.conf) specified by UUID; could this
be causing the spare to be excluded, since it (presumably) does not
have a superblock?

Thanks in advance,
Dave

Re: RAID-5 spare drive: Can't make spare persist across reboot

[Neil Brown]

On Saturday May 30, dave@daverea.com wrote:
>    [First-time poster, done plenty of googling and archive searching
> before sending this, so if it's already been covered my apologies in
> advance...]
> 
> Hello All-
> 
> I am running a 3-disk RAID-5 using kernel 2.6.24 and mdadm 2.6.3 under
> Ubuntu server 8.04LTS. I'm trying to add a spare drive to the array.
> 
> /dev/md0 == /dev/sdc, /dev/sdd, /dev/sda
> 
> I want to add /dev/sdb1 as a spare partition, so I tried `mdadm
> /dev/md0 --add /dev/sdb1` ... this worked great; running `mdadm
> --detail /dev/md0` shows 3 active and 1 spare. Cool!
> 
> However, after a reboot, I'm back to 3 active and 0 spare drives. I
> tried adding the spare to my DEVICES list in /etc/mdadm/mdadm.conf,
> and adding a SPARES=1 attribute to the ARRAY line in that file, still
> no persistence. I tried running `update-initramfs -u` after changing
> mdadm.conf (per an IRC suggestion) but still no persistence.

This should "just work".  There might be some hints in
/var/log/messages if you'd like to post that.

> 
> -- Is it possible to do what I'm trying to do?

Definitely.

> -- Do I need to create the array anew using `mdadm --create` and
> specifying "--spare-devices=1 /dev/sdb1" during creation?

No.

> -- Will creating the array again risk corrupting the data already present?

There is always a risk - you might type the wrong command by mistake.  
Other than that there is no risk.
However doing so won't gain you anything.

> -- Is there any way to instantiate a superblock on /dev/sdb1 so that
> it can be associated with this array's UUID?

You have already done that.  Use
   mdadm --examine /dev/sdb
to confirm.

> -- I have the ARRAY line (in mdadm.conf) specified by UUID; could this
> be causing the spare to be excluded, since it (presumably) does not
> have a superblock?

No.  And it does have a superblock.

NeilBrown

Re: RAID-5 spare drive: Can't make spare persist across reboot

[David Rea]

On Sat, May 30, 2009 at 5:53 PM, Neil Brown <neilb@suse.de> wrote:

>> However, after a reboot, I'm back to 3 active and 0 spare drives. [snip]
>
> This should "just work".  There might be some hints in
> /var/log/messages if you'd like to post that.

Here is the output of `cat /var/log/messages | grep raid`:
http://pastebin.ca/1441575

It seems that raid5 only sees the sd[acd] as active drives, but
doesn't pick up the spare.

>> -- Is there any way to instantiate a superblock on /dev/sdb1 so that
>> it can be associated with this array's UUID?
>
> You have already done that.  Use
>   mdadm --examine /dev/sdb
> to confirm.

This successfully identifies /dev/sdb1 as a spare: http://pastebin.ca/1441576
However running `mdadm --examine /dev/sda` for example does not:
http://pastebin.ca/1441578
And running `mdadm --detail /dev/md0` also says no spares:
http://pastebin.ca/1441579

(let me know if you'd rather I paste these inline, for archival
purposes or what have you)

>> -- I have the ARRAY line (in mdadm.conf) specified by UUID; could this
>> be causing the spare to be excluded, since it (presumably) does not
>> have a superblock?
>
> No.  And it does have a superblock.

Gotcha, I stand corrected! I guess at this point I just need to figure
out how to get the spare to recognize on bootup, since it's already
got the same UUID and is listed in mdadm.conf:
http://pastebin.ca/1441588

Thanks again for your help,
Dave
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID-5 spare drive: Can't make spare persist across reboot

[NeilBrown]

On Sun, May 31, 2009 8:26 am, David Rea wrote:
> On Sat, May 30, 2009 at 5:53 PM, Neil Brown <neilb@suse.de> wrote:
>
>>> However, after a reboot, I'm back to 3 active and 0 spare drives.
>>> [snip]
>>
>> This should "just work".  There might be some hints in
>> /var/log/messages if you'd like to post that.
>
> Here is the output of `cat /var/log/messages | grep raid`:
> http://pastebin.ca/1441575

That isn't very useful.  Getting the whole /var/log/messasges
would be best - then I can see what isn't there as well as what is.
maybe
   grep -C 10 -E '(md|raid)' /var/log/messages
might be enough.

>
> It seems that raid5 only sees the sd[acd] as active drives, but
> doesn't pick up the spare.
>
>>> -- Is there any way to instantiate a superblock on /dev/sdb1 so that
>>> it can be associated with this array's UUID?
>>
>> You have already done that.  Use
>>   mdadm --examine /dev/sdb
>> to confirm.
>
> This successfully identifies /dev/sdb1 as a spare:
> http://pastebin.ca/1441576
> However running `mdadm --examine /dev/sda` for example does not:
> http://pastebin.ca/1441578

That would be because the array was assembled without /dev/sdb1
so even if sda knew about it before, it will have been told to forget.

Is there a reason why you are using whole drives (sda, sdc, sdd) for
some devices, and just a partition (sdb1) for others?




> And running `mdadm --detail /dev/md0` also says no spares:
> http://pastebin.ca/1441579
>
> (let me know if you'd rather I paste these inline, for archival
> purposes or what have you)

Inline is generally easier to work with, unless things are huge.


NeilBrown



>
>>> -- I have the ARRAY line (in mdadm.conf) specified by UUID; could this
>>> be causing the spare to be excluded, since it (presumably) does not
>>> have a superblock?
>>
>> No.  And it does have a superblock.
>
> Gotcha, I stand corrected! I guess at this point I just need to figure
> out how to get the spare to recognize on bootup, since it's already
> got the same UUID and is listed in mdadm.conf:
> http://pastebin.ca/1441588
>
> Thanks again for your help,
> Dave
>

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID-5 spare drive: Can't make spare persist across reboot

[David Rea]

On Sat, May 30, 2009 at 6:44 PM, NeilBrown <neilb@suse.de> wrote:

>> Here is the output of `cat /var/log/messages | grep raid`:
>> http://pastebin.ca/1441575
>
> That isn't very useful.  Getting the whole /var/log/messasges
> would be best - then I can see what isn't there as well as what is.
> maybe
>   grep -C 10 -E '(md|raid)' /var/log/messages
> might be enough.

Here is the output of that command, somewhat lengthy: http://pastebin.ca/1441666
(inline from now on)

> That would be because the array was assembled without /dev/sdb1
> so even if sda knew about it before, it will have been told to forget.

I'm assuming that the array is re-assembled at every boot (per
http://linux-raid.osdl.org/index.php/RAID_setup#Using_the_Array); will
doing an explicit re-assmeble using `mdadm --assemble` correct this on
a "stickier" basis? How can I instruct the kernel to assemble the
array *with* /dev/sdb1 every time? I originally thought this was done
based on /etc/mdadm/mdadm.conf?

> Is there a reason why you are using whole drives (sda, sdc, sdd) for
> some devices, and just a partition (sdb1) for others?

When I originally created this array (2006), it only had the 3 active
drives, and I used the entire devices because I didn't know any
better. When I rebuilt the machine more recently (2008) I added a
larger boot drive, and had plenty of extra space to reserve a
partition to use as a spare for the array.

I happen to have several 1TiB SATA drives (awaiting build into a
different RAID machine) sitting around, so I could back up this array
and re-create it if that's the only way we can make the spare
persistent.

Thanks,
Dave
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html