s/w raid and bios renumbering HDs

s/w raid and bios renumbering HDs

[Hari Bhaskaran]

Hi,

I am trying to setup a RAID-1 setup for the boot/root partition. I got
the setup working, except what I see with some of my tests leave me
less convinced that it is actually working. My system is debian 3.1
and I am not using the raid-setup options in the debian-installer,
I am trying to add raid-1 to an existing system (followed
http://www.tldp.org/HOWTO/Software-RAID-HOWTO.html  -- 7.4 method 2)

I have /dev/hda (master on primary) and /dev/hdc (master on secondary)
setup as mirrors. I also have a cdrom on /dev/hdd. Now if I disconnect
hda and reboot, everything seems work - except what used to be
/dev/hdc comes up as /dev/hda. I know this since I the bios does
complain that "primary disk 0" is missing and I would have expected a
missing hda, not a missing hdc. Anyways, the software seems to
recognize the "failed-disk" fine if I connect the real hda back. Is
this the way it is supposed to work? Can I rely on this? Also what
happens when I move on to fancier setups like raid5?. My box is a dell
400sc with some phoenix bios (doesnt have many options either). I get
different (still unexpected) results with the cdrom connected and not.

Question #2 (probably related to my problem)

My grub menu.lst is as follows (/dev/md0 is made of /dev/hda1 and /dev/hdc1)
For testing, I made two entries (one for (hd0,0) and another for
(hd1,0)). The howto
I was reading wasn't clear to me. Should I be making just one entry
pointing to /dev/md0?
Also trying labels for "hda" and "hdc" after connecting the faulty
drive back gave me different results ( in one case I was looking at
"older" data and in the other case I wasn't)

(ignore the vs2.1.xxx. it is a linux-vserver patch - shouldn't matter here)

title           Debian GNU/Linux, kernel 2.6.13.3-vs2.1.0-rc4-RAID-hda
root            (hd0,0)
kernel          /boot/vmlinuz-2.6.13.3-vs2.1.0-rc4 root=/dev/md0 ro
initrd          /boot/initrd.img-2.6.13.3-vs2.1.0-rc4.md0
savedefault
boot

title           Debian GNU/Linux, kernel 2.6.13.3-vs2.1.0-rc4-RAID-hdc
root            (hd1,0)
kernel          /boot/vmlinuz-2.6.13.3-vs2.1.0-rc4 root=/dev/md0 ro
initrd          /boot/initrd.img-2.6.13.3-vs2.1.0-rc4.md0
savedefault
boot

Any help is appreciated. If there is a better/current HOWTO, please
let me know. The ones I have seen so far refer to now deprecated tools
(raidtools or raidtools2) and I have had a hard time trying to find
the equivalent syntax for mdadm.

--
Hari

RE: s/w raid and bios renumbering HDs

[Callahan, Tom]

You are testing failover with reboots. So when Linux probes the disks, it is
putting "hdc" where "hda" used to be.... This seems a bit strange, as
hda/hdb should theoretically be IDE1 and hdc/hdd should be IDE2....

As far as your grub setup, it looks perfectly fine. You should have two
entries as you have, because if disc1 fails, you cannot boot to hd(0,0) and
vice-versa.
One gotcha, make sure grub is installed in the MBR of BOTH drives, not just
the MD device....

Thanks,
Tom Callahan
TESSCO Technologies Inc.
410-229-1361


-----Original Message-----
From: linux-raid-owner@vger.kernel.org
[mailto:linux-raid-owner@vger.kernel.org]On Behalf Of Hari Bhaskaran
Sent: Monday, October 31, 2005 10:57 AM
To: linux-raid@vger.kernel.org
Subject: s/w raid and bios renumbering HDs


Hi,

I am trying to setup a RAID-1 setup for the boot/root partition. I got
the setup working, except what I see with some of my tests leave me
less convinced that it is actually working. My system is debian 3.1
and I am not using the raid-setup options in the debian-installer,
I am trying to add raid-1 to an existing system (followed
http://www.tldp.org/HOWTO/Software-RAID-HOWTO.html  -- 7.4 method 2)

I have /dev/hda (master on primary) and /dev/hdc (master on secondary)
setup as mirrors. I also have a cdrom on /dev/hdd. Now if I disconnect
hda and reboot, everything seems work - except what used to be
/dev/hdc comes up as /dev/hda. I know this since I the bios does
complain that "primary disk 0" is missing and I would have expected a
missing hda, not a missing hdc. Anyways, the software seems to
recognize the "failed-disk" fine if I connect the real hda back. Is
this the way it is supposed to work? Can I rely on this? Also what
happens when I move on to fancier setups like raid5?. My box is a dell
400sc with some phoenix bios (doesnt have many options either). I get
different (still unexpected) results with the cdrom connected and not.

Question #2 (probably related to my problem)

My grub menu.lst is as follows (/dev/md0 is made of /dev/hda1 and /dev/hdc1)
For testing, I made two entries (one for (hd0,0) and another for
(hd1,0)). The howto
I was reading wasn't clear to me. Should I be making just one entry
pointing to /dev/md0?
Also trying labels for "hda" and "hdc" after connecting the faulty
drive back gave me different results ( in one case I was looking at
"older" data and in the other case I wasn't)

(ignore the vs2.1.xxx. it is a linux-vserver patch - shouldn't matter here)

title           Debian GNU/Linux, kernel 2.6.13.3-vs2.1.0-rc4-RAID-hda
root            (hd0,0)
kernel          /boot/vmlinuz-2.6.13.3-vs2.1.0-rc4 root=/dev/md0 ro
initrd          /boot/initrd.img-2.6.13.3-vs2.1.0-rc4.md0
savedefault
boot

title           Debian GNU/Linux, kernel 2.6.13.3-vs2.1.0-rc4-RAID-hdc
root            (hd1,0)
kernel          /boot/vmlinuz-2.6.13.3-vs2.1.0-rc4 root=/dev/md0 ro
initrd          /boot/initrd.img-2.6.13.3-vs2.1.0-rc4.md0
savedefault
boot

Any help is appreciated. If there is a better/current HOWTO, please
let me know. The ones I have seen so far refer to now deprecated tools
(raidtools or raidtools2) and I have had a hard time trying to find
the equivalent syntax for mdadm.

--
Hari
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: s/w raid and bios renumbering HDs

[dean gaudet]

On Mon, 31 Oct 2005, Hari Bhaskaran wrote:

> Hi,
> 
> I am trying to setup a RAID-1 setup for the boot/root partition. I got
> the setup working, except what I see with some of my tests leave me
> less convinced that it is actually working. My system is debian 3.1
> and I am not using the raid-setup options in the debian-installer,
> I am trying to add raid-1 to an existing system (followed
> http://www.tldp.org/HOWTO/Software-RAID-HOWTO.html  -- 7.4 method 2)

fyi there's a debian-specific doc at 
/usr/share/doc/mdadm/rootraiddoc.97.html which i've always found useful.


> I have /dev/hda (master on primary) and /dev/hdc (master on secondary)
> setup as mirrors. I also have a cdrom on /dev/hdd. Now if I disconnect
> hda and reboot, everything seems work - except what used to be
> /dev/hdc comes up as /dev/hda. I know this since I the bios does
> complain that "primary disk 0" is missing and I would have expected a
> missing hda, not a missing hdc.

huh i wonder if the bios has tweaked the ide controller to swap the 
primary/secondary somehow -- probably cuts down on support calls for 
people who plug things in wrong.  there could be a bios option to stop 
this swapping.


> Anyways, the software seems to
> recognize the "failed-disk" fine if I connect the real hda back. Is
> this the way it is supposed to work? Can I rely on this? Also what
> happens when I move on to fancier setups like raid5?.

the md superblock (at the end of the partition) contains reconstruction
information and UUIDs... the device names they end up on are mostly
irrelevant if you've got things configured properly.  i've moved disks
between /dev/hd* and /dev/sd* going from pata controllers to 3ware
controllers with no problem.

for raids other than the root raid you pretty much want to edit
/etc/mdadm/mdadm.conf and make sure it has "DEVICE partitions" and has
ARRAY entries for each of your arrays listing the UUID.  you can generate
these entries with "mdadm --detail --scan" (see examples on man page).
you can plug the non-root disks in any way you want and things will still
work if you've configured this.

the root is the only one which you need to be careful with -- when debian 
installs your kernel it constructs an initrd which lists the minimum 
places it will search for the root raid components... for example on one 
of my boxes:

# mkdir /mnt/cramfs
# mount -o ro,loop /boot/initrd.img-2.6.13-1-686-smp /mnt/cramfs
# cat /mnt/cramfs/script
ROOT=/dev/md3
mdadm -A /dev/md3 -R -u 2b3a5b77:c7b4ab81:a2b8322a:db5c4e88 /dev/sdb4 /dev/sda4
# umount /mnt/cramfs

it's only expecting to look for the root raid components in those two
partitions... seems kind of unfortunate really 'cause the script could
be configured to look in any partition.

in theory you can hand-edit the initrd if you plan to move root disks to
another position... you can't mount a cramfs rw, so you need to mount,
copy, edit, and run mkcramfs ... and i suggest not deleting your original
initrd, and i suggest copy&pasting the /boot/grub/menu.lst entries to give
you the option of booting the old initrd or your new made-by-hand one.


> title           Debian GNU/Linux, kernel 2.6.13.3-vs2.1.0-rc4-RAID-hda
> root            (hd0,0)
> kernel          /boot/vmlinuz-2.6.13.3-vs2.1.0-rc4 root=/dev/md0 ro
> initrd          /boot/initrd.img-2.6.13.3-vs2.1.0-rc4.md0
> savedefault
> boot
> 
> title           Debian GNU/Linux, kernel 2.6.13.3-vs2.1.0-rc4-RAID-hdc
> root            (hd1,0)
> kernel          /boot/vmlinuz-2.6.13.3-vs2.1.0-rc4 root=/dev/md0 ro
> initrd          /boot/initrd.img-2.6.13.3-vs2.1.0-rc4.md0
> savedefault
> boot

i don't think you need both.  when your first disk is dead the bios
shifts the second disk forward... and hd0 / hd1 refer to bios ordering.
i don't have both in my configs, but then i haven't bothered testing
booting off the second disk in a long time.  (i always have live-cds
such as knoppix handy for fixing boot problems.)

-dean

Re: s/w raid and bios renumbering HDs

[Hari Bhaskaran]

on 10/31/2005 11:47 AM dean gaudet said the following:
> huh i wonder if the bios has tweaked the ide controller to swap the 
> primary/secondary somehow -- probably cuts down on support calls for 
> people who plug things in wrong.  there could be a bios option to stop 
> this swapping.
>   
Not that I can see. Cheap dell machines like the one I have come with a 
single-page bios
setup screen. However, now I think it may not all be the bios. I have 
two more HDs connected
with an Adaptec pci-ide controller (which I plan to use for a raid-5). I 
will do some more experiments
so that I understand how this whole thing works.

> for raids other than the root raid you pretty much want to edit
> /etc/mdadm/mdadm.conf and make sure it has "DEVICE partitions" and has
>   
So that "DEVICE paritions" line was really supposed to be there? Hehe... 
I thought it was just a
help message and replaced it with "DEVICE /dev/hda1 /dev/hdc1" :)
> ARRAY entries for each of your arrays listing the UUID.  you can generate
> these entries with "mdadm --detail --scan" (see examples on man page).
>   
it took me a whole Sunday to figure that one out. I followed the howto, 
created the array and didn't know
how to start it back up after a reboot ( I hadn't asked mdadm to start 
raid devices at system startup). Even
when I did that, the array didn't come up until I put those ARRAY lines. 
I kept on editing raidtab - I hadn't read
the tiny print in mdadm manpage.

If I ever end up in a situation with a non-root raid down (say I did 
--stop), how do I start it back up? (--run seems
to give me some errors). Anyways, more "rtfm" to do.

> you can plug the non-root disks in any way you want and things will still
> work if you've configured this.
>   
That is nice.
> the root is the only one which you need to be careful with -- when debian 
> installs your kernel it constructs an initrd which lists the minimum 
> places it will search for the root raid components... for example on one 
> of my boxes:
>
> # mkdir /mnt/cramfs
> # mount -o ro,loop /boot/initrd.img-2.6.13-1-686-smp /mnt/cramfs
> # cat /mnt/cramfs/script
> ROOT=/dev/md3
> mdadm -A /dev/md3 -R -u 2b3a5b77:c7b4ab81:a2b8322a:db5c4e88 /dev/sdb4 /dev/sda4
> # umount /mnt/cramfs
>   
Did u install yours with raid options in the debian installer? I dont 
think my initrd image would have all these ( I dont have
access to the machine now to check) - but I wouldn't think the mkinitrd 
that I used to created the initrd image would
know that I am using raid or not ( I am talking about the mdadm 
references in your script). Or are you saying you
added these yourself?
> it's only expecting to look for the root raid components in those two
> partitions... seems kind of unfortunate really 'cause the script could
> be configured to look in any partition.
>
> in theory you can hand-edit the initrd if you plan to move root disks to
> another position... you can't mount a cramfs rw, so you need to mount,
> copy, edit, and run mkcramfs ... and i suggest not deleting your original
> initrd, and i suggest copy&pasting the /boot/grub/menu.lst entries to give
> you the option of booting the old initrd or your new made-by-hand one.
>
>
>   
>> title           Debian GNU/Linux, kernel 2.6.13.3-vs2.1.0-rc4-RAID-hda
>> root            (hd0,0)
>> kernel          /boot/vmlinuz-2.6.13.3-vs2.1.0-rc4 root=/dev/md0 ro
>> initrd          /boot/initrd.img-2.6.13.3-vs2.1.0-rc4.md0
>> savedefault
>> boot
>>
>> title           Debian GNU/Linux, kernel 2.6.13.3-vs2.1.0-rc4-RAID-hdc
>> root            (hd1,0)
>> kernel          /boot/vmlinuz-2.6.13.3-vs2.1.0-rc4 root=/dev/md0 ro
>> initrd          /boot/initrd.img-2.6.13.3-vs2.1.0-rc4.md0
>> savedefault
>> boot
>>     
>
> i don't think you need both.  when your first disk is dead the bios
> shifts the second disk forward... and hd0 / hd1 refer to bios ordering.
> i don't have both in my configs, but then i haven't bothered testing
> booting off the second disk in a long time.  (i always have live-cds
> such as knoppix handy for fixing boot problems.)
>
> -dean
>
>   

--
Hari

Re: s/w raid and bios renumbering HDs

[dean gaudet]

On Mon, 31 Oct 2005, Hari Bhaskaran wrote:

> So that "DEVICE paritions" line was really supposed to be there? Hehe... I
> thought it was just a
> help message and replaced it with "DEVICE /dev/hda1 /dev/hdc1" :)

you can use "DEVICE /dev/hda1 /dev/hdc1" ... but then mdadm scans will 
only consider those two partitions... if you use "DEVICE partitions" it'll 
look at all detected partitions for the components.  it makes it easy when 
you move disks around to new controllers and their location changes, 
things will continue to jfw.


> If I ever end up in a situation with a non-root raid down (say I did --stop),
> how do I start it back up? (--run seems
> to give me some errors). Anyways, more "rtfm" to do.

you want --assemble ...


> > the root is the only one which you need to be careful with -- when debian
> > installs your kernel it constructs an initrd which lists the minimum places
> > it will search for the root raid components... for example on one of my
> > boxes:
> > 
> > # mkdir /mnt/cramfs
> > # mount -o ro,loop /boot/initrd.img-2.6.13-1-686-smp /mnt/cramfs
> > # cat /mnt/cramfs/script
> > ROOT=/dev/md3
> > mdadm -A /dev/md3 -R -u 2b3a5b77:c7b4ab81:a2b8322a:db5c4e88 /dev/sdb4
> > /dev/sda4
> > # umount /mnt/cramfs
> >   
> Did u install yours with raid options in the debian installer? I dont think my
> initrd image would have all these ( I dont have
> access to the machine now to check) - but I wouldn't think the mkinitrd that I
> used to created the initrd image would
> know that I am using raid or not ( I am talking about the mdadm references in
> your script). Or are you saying you
> added these yourself?

there's really no reason to avoid the debian installer's raid support if 
you know you want raid, but i haven't used it much.

you only need to do initrd edits by hand once if you're converting root to 
a raid. there's a few steps in the debian doc about this (see Part II. 
RAID using initrd and grub) /usr/share/doc/mdadm/rootraiddoc.97.html.

after that initial change, and you've managed to boot with root on raid 
then subsequent mkinitrd *should* fill in the details automatically... 
i.e. every time you upgrade your kernel you get a new initrd, and it 
should automatically include the root raid setup.

-dean