What will happen with spares in this scenario?

What will happen with spares in this scenario?

[John McNulty]

Hello,

I have 16 disks and two RAID controllers, 8 disks per controller.
These are c0d0 -> c0d7 and c1d0 -> c1d7.    The first disk on each
controller is configured with two partitions:  300MB (p1) and the rest
of the disk (p2).   Then I make some mirror plexes:

c0d0p1 + c1d0p1 = md0
c0d0p2 + c1d0p2 = md1
c0d1p1 + c1d1p1 = md2
c0d2p1 + c1d2p1 = md3
c0d3p1 + c1d3p1 = md4
c0d4p1 + c1d4p1 = md5
c0d5p1 + c1d5p1 = md6
c0d6p1 + c1d6p1 = md7

Disks c0d7p1 and c1d7p1 are added to md7 as spares, and all the
mirrors are configured to be in the same spare-group in mdadm.conf.

If I loose a disk in a mirror other than md7, then md will steal a
spare disk from md7 and use that.  But what if the disk is c0d0 or
c1d0?  There are two mirror plexes sitting on top of these disks
mirroring different partitions.  Will md work this out, partition the
spares accordingly and fix up both using one spare disk, or will it
just break, or will it get really confused and try to use both the
spares in md7 to resolve the situation?

I think I already know what the answer's going to be and that I've
just shot myself in the foot?  Well, not quite as this isn't a live
server yet so I have time to correct this.

Rgds,

John

Re: What will happen with spares in this scenario?

[Goswin von Brederlow]

John McNulty <johnmcn1@googlemail.com> writes:

> Hello,
>
> I have 16 disks and two RAID controllers, 8 disks per controller.
> These are c0d0 -> c0d7 and c1d0 -> c1d7.    The first disk on each
> controller is configured with two partitions:  300MB (p1) and the rest
> of the disk (p2).   Then I make some mirror plexes:
>
> c0d0p1 + c1d0p1 = md0
> c0d0p2 + c1d0p2 = md1
> c0d1p1 + c1d1p1 = md2
> c0d2p1 + c1d2p1 = md3
> c0d3p1 + c1d3p1 = md4
> c0d4p1 + c1d4p1 = md5
> c0d5p1 + c1d5p1 = md6
> c0d6p1 + c1d6p1 = md7
>
> Disks c0d7p1 and c1d7p1 are added to md7 as spares, and all the
> mirrors are configured to be in the same spare-group in mdadm.conf.
>
> If I loose a disk in a mirror other than md7, then md will steal a
> spare disk from md7 and use that.  But what if the disk is c0d0 or
> c1d0?  There are two mirror plexes sitting on top of these disks
> mirroring different partitions.  Will md work this out, partition the
> spares accordingly and fix up both using one spare disk, or will it
> just break, or will it get really confused and try to use both the
> spares in md7 to resolve the situation?
>
> I think I already know what the answer's going to be and that I've
> just shot myself in the foot?  Well, not quite as this isn't a live
> server yet so I have time to correct this.

Looks like it. I think it will use c0d7p1 to fix md0 and c1d7p1 to fix
md1. So drop md1, make c0d0p1 as big as the others and then partition
md0 eigther directly or via lvm.

Or write your own handler to add sparse to failing devices that knows
when it needs to partition.

> Rgds,
>
> John

MfG
        Goswin

Re: What will happen with spares in this scenario?

[John McNulty]

On 29 Jun 2009, at 17:08, Goswin von Brederlow wrote:

> So drop md1, make c0d0p1 as big as the others and then partition
> md0 eigther directly or via lvm.

You wouldn't happen to know if it's possible to create a partitionable  
array like this in Kickstart would you?   I've had a sniff around  
Redhat's docs and a documented way of doing this isn't jumping out at  
me.    If I had to guess then it might look something like this?

part  pv.1  --size 1 --ondisk cciss/c0d0 --grow --asprimary
part  pv.2  --size 1 --ondisk cciss/c1d0 --grow --asprimary
raid  pv.3  --level=RAID1 --device=md_d0 pv.1 pv.2
part /boot --fstype ext3 --size 300 --ondisk md_d0
part pv.4 --fstype "physical volume (LVM) --size 1 --ondisk md_d0 --grow
volgroup system --pesize=32768 pv.4
logvol / --fstype ext3 --name=root --vgname=system --size=51200
logvol swap --fstype swap --name=swap --vgname=system --size=8192

This might be a bit ambitious, and I can't see an option to "raid"  
that would tell it to create a partitionable raid device, i.e. run -- 
auto=mdp behind the scenes.   Unless it takes a hint from the device  
name and does that without asking?

The systems I'm working on are to be kickstarted from a Spacewalk  
server.

Rgds,

John

Re: What will happen with spares in this scenario?

[Goswin von Brederlow]

John McNulty <johnmcn1@googlemail.com> writes:

> On 29 Jun 2009, at 17:08, Goswin von Brederlow wrote:
>
>
>           So drop md1, make c0d0p1 as big as the others and then
>      partition
>      md0 eigther directly or via lvm.
>
>
>
>
> You wouldn't happen to know if it's possible to create a partitionable array
> like this in Kickstart would you?   I've had a sniff around Redhat's docs and
> a documented way of doing this isn't jumping out at me.    If I had to guess
> then it might look something like this?

Totally Debian here.

> part  pv.1  --size 1 --ondisk cciss/c0d0 --grow --asprimary
>
> part  pv.2  --size 1 --ondisk cciss/c1d0 --grow --asprimary
>
> raid  pv.3  --level=RAID1 --device=md_d0 pv.1 pv.2
>
>
> part /boot --fstype ext3 --size 300 --ondisk md_d0
>
> part pv.4 --fstype "physical volume (LVM) --size 1 --ondisk md_d0 --grow
>
> volgroup system --pesize=32768 pv.4
> logvol / --fstype ext3 --name=root --vgname=system --size=51200
> logvol swap --fstype swap --name=swap --vgname=system --size=8192

Hmm, do I understand this right that you want 2 partitions. One for
/boot and one for everything else on lvm?

Just a crazy idea. But why not put /boot on lvm too and use grub2 or
lilo as bootloader? Or does Kickstart not allow that?

> This might be a bit ambitious, and I can't see an option to "raid" that would
> tell it to create a partitionable raid device, i.e. run --auto=mdp behind the
> scenes.   Unless it takes a hint from the device name and does that without
> asking?
>
>
>
> The systems I'm working on are to be kickstarted from a Spacewalk server.
>
>
>
> Rgds,
>
>
>
> John

MfG
        Goswin
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: What will happen with spares in this scenario?

[John McNulty]

On Tue, Jun 30, 2009 at 5:26 AM, Goswin von Brederlow<goswin-v-b@web.de> wrote:
>
> Hmm, do I understand this right that you want 2 partitions. One for
> /boot and one for everything else on lvm?
>
> Just a crazy idea. But why not put /boot on lvm too and use grub2 or
> lilo as bootloader? Or does Kickstart not allow that?

I didn't think that booting from an LVM device was supported, whether
it's sitting on an md raid device or not.  How is the bootloader meant
to find the kernel and initrd without first constructing the VG and
mounting the /boot Logical Volume.  It's been quite a while since I
looked (we get into the habit of doing things the same way) but last
time I checked this didn't work.

Rgds,

John

Re: What will happen with spares in this scenario?

[Goswin von Brederlow]

John McNulty <johnmcn1@googlemail.com> writes:

> On Tue, Jun 30, 2009 at 5:26 AM, Goswin von Brederlow<goswin-v-b@web.de> wrote:
>>
>> Hmm, do I understand this right that you want 2 partitions. One for
>> /boot and one for everything else on lvm?
>>
>> Just a crazy idea. But why not put /boot on lvm too and use grub2 or
>> lilo as bootloader? Or does Kickstart not allow that?
>
> I didn't think that booting from an LVM device was supported, whether
> it's sitting on an md raid device or not.  How is the bootloader meant
> to find the kernel and initrd without first constructing the VG and
> mounting the /boot Logical Volume.  It's been quite a while since I
> looked (we get into the habit of doing things the same way) but last
> time I checked this didn't work.
>
> Rgds,
>
> John

In case of grub2: It does support both raid and lvm. The raid
superblocks are parsed to construct mdX devices and the lvm metadata
is parsed to locate lvm logical volumes. So you could say it does
construct the LV and mounts the FS.

In case of lilo: Lilo only stores a list of blocks where the
kernel/initrd are on the device. Afaik each component device of a raid
stores the the block numbers for the kernel/initrd on that component
device. So no matter which component device boots it will find the
kernel/initrd on the same device. The raid1 and lvm are completly
circumvented.

MfG
        Goswin

Re: What will happen with spares in this scenario?

[John McNulty]

On 30 Jun 2009, at 18:54, Goswin von Brederlow wrote:

> In case of grub2: It does support both raid and lvm. The raid
> superblocks are parsed to construct mdX devices and the lvm metadata
> is parsed to locate lvm logical volumes. So you could say it does
> construct the LV and mounts the FS.

Interesting, thanks.  I've not been following grub2 development.   
Given the production nature of these systems though (and the customer)  
I'm stuck with grub legacy until Redhat update it and support grub2,  
which judging from the chatter on the Fedora Project Portal about the  
possibility of including it in Fedora 12, could be some years off.   
Looks like Ubuntu will be putting it into 9.10 though, so I'll have a  
play with it on one of my dev boxes at home.

> In case of lilo: Lilo only stores a list of blocks where the
> kernel/initrd are on the device. Afaik each component device of a raid
> stores the the block numbers for the kernel/initrd on that component
> device. So no matter which component device boots it will find the
> kernel/initrd on the same device. The raid1 and lvm are completly
> circumvented.

Have not touched lilo since grub went main stream, but will have  
another look at that.

I've gone over all this with the customer.  They want belt and braces  
protection (they are to be very critical systems in a hospital) so  
we've decided to hardware mirror c0d0 and c0d1 for the disks on one  
controller, plus the same for c1d0 and c1d1 on the second controller,  
then partition and md mirror both of those.  The rest of the disks  
will be handled separately and the kickstart config is quite simple.   
It's a bit overkill, but it won't be failing disks or controllers that  
cause the systems to fail in the future.

Rgds,

John

Re: What will happen with spares in this scenario?

[Goswin von Brederlow]

John McNulty <johnmcn1@googlemail.com> writes:

> On 30 Jun 2009, at 18:54, Goswin von Brederlow wrote:
>
>> In case of grub2: It does support both raid and lvm. The raid
>> superblocks are parsed to construct mdX devices and the lvm metadata
>> is parsed to locate lvm logical volumes. So you could say it does
>> construct the LV and mounts the FS.
>
> Interesting, thanks.  I've not been following grub2 development.
> Given the production nature of these systems though (and the customer)
> I'm stuck with grub legacy until Redhat update it and support grub2,
> which judging from the chatter on the Fedora Project Portal about the
> possibility of including it in Fedora 12, could be some years off.
> Looks like Ubuntu will be putting it into 9.10 though, so I'll have a
> play with it on one of my dev boxes at home.
>
>> In case of lilo: Lilo only stores a list of blocks where the
>> kernel/initrd are on the device. Afaik each component device of a raid
>> stores the the block numbers for the kernel/initrd on that component
>> device. So no matter which component device boots it will find the
>> kernel/initrd on the same device. The raid1 and lvm are completly
>> circumvented.
>
> Have not touched lilo since grub went main stream, but will have
> another look at that.
>
> I've gone over all this with the customer.  They want belt and braces
> protection (they are to be very critical systems in a hospital) so
> we've decided to hardware mirror c0d0 and c0d1 for the disks on one
> controller, plus the same for c1d0 and c1d1 on the second controller,
> then partition and md mirror both of those.  The rest of the disks
> will be handled separately and the kickstart config is quite simple.
> It's a bit overkill, but it won't be failing disks or controllers that
> cause the systems to fail in the future.
>
> Rgds,
>
> John

Don't worry, when you have a heart attack it won't be the disk that
fails. Some 0.1c diode ion the motherboard will burn through and take
out the system.

MfG
        Goswin