softraid and multiple distros

softraid and multiple distros

[Dexter Filmore]

What do I need to do when I want to install a different distro on the machine 
with a raid5 array?
Which files do I need? /etc/mdadm.conf? /etc/raittab? both?

Dex


-- 
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCS d--(+)@ s-:+ a- C+++(++++) UL+>++++ P+>++ L+++>++++ E-- W++ N o? K-
w--(---) !O M+ V- PS++(+) PE(-) Y++ PGP t++(---)@ 5 X+(++) R+(++) tv--(+)@ 
b++(+++) DI+++ D G++ e* h>++ r%>* y?
------END GEEK CODE BLOCK------

http://www.stop1984.com
http://www.againsttcpa.com

Re: softraid and multiple distros

[Mark Hahn]

> What do I need to do when I want to install a different distro on the machine 
> with a raid5 array?
> Which files do I need? /etc/mdadm.conf? /etc/raittab? both?

MD doesn't need any files to function, since it can auto-assemble
arrays based on their superblocks (for partition-type 0xfd).

Re: softraid and multiple distros

[Dexter Filmore]

Am Sonntag, 14. Mai 2006 16:50 schrieben Sie:
> > What do I need to do when I want to install a different distro on the
> > machine with a raid5 array?
> > Which files do I need? /etc/mdadm.conf? /etc/raittab? both?
>
> MD doesn't need any files to function, since it can auto-assemble
> arrays based on their superblocks (for partition-type 0xfd).

I see. Now an issue arises someone else here mentioned: 
My first attempt was to use the entire disks, then I was hinted that this 
approach wasn't too hot so I made partitions. 
Now the devices have all two superblocks, the one left from the first try 
which are now kinda orphaned and those now active. 
Can I trust mdadm to handle this properly on its own?

Dex


-- 
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCS d--(+)@ s-:+ a- C+++(++++) UL+>++++ P+>++ L+++>++++ E-- W++ N o? K-
w--(---) !O M+ V- PS++(+) PE(-) Y++ PGP t++(---)@ 5 X+(++) R+(++) tv--(+)@ 
b++(+++) DI+++ D G++ e* h>++ r%>* y?
------END GEEK CODE BLOCK------

http://www.stop1984.com
http://www.againsttcpa.com

Re: softraid and multiple distros

[Mark Hahn]

> Now the devices have all two superblocks, the one left from the first try 
> which are now kinda orphaned and those now active. 
> Can I trust mdadm to handle this properly on its own?

I'm not sure what "properly" means.  you should not leave around 0xfd 
partitions with bogus superblocks, since MD will certainly try to 
assemble them.  I don't know offhand how it decides which components 
to make into a single array (UUID?), but why screw around?
for "orphan" partitions, either change the partition type 
or zero the superblock or both...

Re: softraid and multiple distros

[Dexter Filmore]

Am Sonntag, 14. Mai 2006 20:42 schrieben Sie:
> > Now the devices have all two superblocks, the one left from the first try
> > which are now kinda orphaned and those now active.
> > Can I trust mdadm to handle this properly on its own?
>
> I'm not sure what "properly" means.  you should not leave around 0xfd

Well, properly means properly. As opposite of "f!ck up".
Here: chose the superblocks from the partitions instead of from the entire 
hdd.
There is everything fine with those fs partitions and the array, but there 
seems some old superblocks lying to be around behind those partitions.

> or zero the superblock or both...

More questions: is the raid superblock the same as an "ordinary" file system 
superblock? 
"Zero" the superblock - the orphaned one, I assume? This is not like "zero it 
and linux makes a new one" or so?

Dex


-- 
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCS d--(+)@ s-:+ a- C+++(++++) UL+>++++ P+>++ L+++>++++ E-- W++ N o? K-
w--(---) !O M+ V- PS++(+) PE(-) Y++ PGP t++(---)@ 5 X+(++) R+(++) tv--(+)@ 
b++(+++) DI+++ D G++ e* h>++ r%>* y?
------END GEEK CODE BLOCK------

http://www.stop1984.com
http://www.againsttcpa.com

Re: softraid and multiple distros

[Mark Hahn]

> > or zero the superblock or both...
> 
> More questions: is the raid superblock the same as an "ordinary" file system 
> superblock? 

iirc, the raid SB is at the end, and most FS's base theirs at the beginning.

> "Zero" the superblock - the orphaned one, I assume? This is not like "zero it 
> and linux makes a new one" or so?

zero it so that you're not presenting inconsistent stuff to mdadm
when it tries to auto-assemble.

Re: softraid and multiple distros

[Neil Brown]

On Sunday May 14, Dexter.Filmore@gmx.de wrote:
> Am Sonntag, 14. Mai 2006 16:50 schrieben Sie:
> > > What do I need to do when I want to install a different distro on the
> > > machine with a raid5 array?
> > > Which files do I need? /etc/mdadm.conf? /etc/raittab? both?
> >
> > MD doesn't need any files to function, since it can auto-assemble
> > arrays based on their superblocks (for partition-type 0xfd).
> 
> I see. Now an issue arises someone else here mentioned: 
> My first attempt was to use the entire disks, then I was hinted that this 
> approach wasn't too hot so I made partitions. 

I always use entire disks if I want the entire disks raided (sounds
obvious, doesn't it...)  I only use partitions when I want to vary the
raid layout for different parts of the disk (e.g. mirrored root, mirrored
swap, raid6 for the rest).   But that certainly doesn't mean it is
wrong to use partitions for the whole disk.


> Now the devices have all two superblocks, the one left from the first try 
> which are now kinda orphaned and those now active. 
> Can I trust mdadm to handle this properly on its own?

You can tell mdadm where to look.  If you want to be sure that it
won't look at entire drives, only partitions, then a line like
   DEVICES /dev/[hs]d*[0-1]
in /etc/mdadm.conf might be what you want. 
However as you should be listing the uuids in /etc/mdadm.conf, any
superblock with an unknown uuid will easily be ignored.

If you are relying nf 0xfd autodetect to assemble your arrays, then
obviously the entire-disk superblock will be ignored (because they
wont be in the right place in any partition).

NeilBrown

Re: softraid and multiple distros

[Dexter Filmore]

> I always use entire disks if I want the entire disks raided (sounds
> obvious, doesn't it...)  I only use partitions when I want to vary the
> raid layout for different parts of the disk (e.g. mirrored root, mirrored
> swap, raid6 for the rest).   But that certainly doesn't mean it is
> wrong to use partitions for the whole disk.

The idea behind this is: let's say a disk fails, and you get a replacement, 
but it has a different geometry or a few blocks less - won't work. 
Even the same disk model might vary after a while.
So I made 0xfd partitions of the size (whole disk minus few megs).

>
> > Now the devices have all two superblocks, the one left from the first try
> > which are now kinda orphaned and those now active.
> > Can I trust mdadm to handle this properly on its own?
>
> You can tell mdadm where to look.  If you want to be sure that it
> won't look at entire drives, only partitions, then a line like
>    DEVICES /dev/[hs]d*[0-1]
> in /etc/mdadm.conf might be what you want.
> However as you should be listing the uuids in /etc/mdadm.conf, any

Umm... yeah, should I?

> superblock with an unknown uuid will easily be ignored.
>
> If you are relying nf 0xfd autodetect to assemble your arrays, then
> obviously the entire-disk superblock will be ignored (because they
> wont be in the right place in any partition).

So mdadm --assemble --scan is fine for my scenario even with those orphaned 
superblocks.

Should get me some sedatives for the day when this all explodes :P

-- 
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCS d--(+)@ s-:+ a- C+++(++++) UL+>++++ P+>++ L+++>++++ E-- W++ N o? K-
w--(---) !O M+ V- PS++(+) PE(-) Y++ PGP t++(---)@ 5 X+(++) R+(++) tv--(+)@ 
b++(+++) DI+++ D G++ e* h>++ r%>* y?
------END GEEK CODE BLOCK------

http://www.stop1984.com
http://www.againsttcpa.com

Re: softraid and multiple distros

[Neil Brown]

On Monday May 15, Dexter.Filmore@gmx.de wrote:
> > I always use entire disks if I want the entire disks raided (sounds
> > obvious, doesn't it...)  I only use partitions when I want to vary the
> > raid layout for different parts of the disk (e.g. mirrored root, mirrored
> > swap, raid6 for the rest).   But that certainly doesn't mean it is
> > wrong to use partitions for the whole disk.
> 
> The idea behind this is: let's say a disk fails, and you get a replacement, 
> but it has a different geometry or a few blocks less - won't work. 
> Even the same disk model might vary after a while.
> So I made 0xfd partitions of the size (whole disk minus few megs).
> 

An alternative is to use the --size option of mdadm to make the array
slightly smaller than the smallest drive.  So you don't need
partitions for this (though it is perfectly alright to use them if you
like). 

> > You can tell mdadm where to look.  If you want to be sure that it
> > won't look at entire drives, only partitions, then a line like
> >    DEVICES /dev/[hs]d*[0-1]
> > in /etc/mdadm.conf might be what you want.
> > However as you should be listing the uuids in /etc/mdadm.conf, any
> 
> Umm... yeah, should I?

What else would you use to uniquely identify the arrays?  Not device
names I hope.

> 
> > superblock with an unknown uuid will easily be ignored.
> >
> > If you are relying nf 0xfd autodetect to assemble your arrays, then
> > obviously the entire-disk superblock will be ignored (because they
> > wont be in the right place in any partition).
> 
> So mdadm --assemble --scan is fine for my scenario even with those orphaned 
> superblocks.

I cannot say for sure without seeing your mdadm.conf, but probably.

> 
> Should get me some sedatives for the day when this all explodes :P

Just make sure it happens on your day off, then someone else will
need the sedatives :-)

NeilBrown

Re: softraid and multiple distros

[Dexter Filmore]

> An alternative is to use the --size option of mdadm to make the array
> slightly smaller than the smallest drive.  

timtowtdi, as usual

> > > However as you should be listing the uuids in /etc/mdadm.conf, any
> > Umm... yeah, should I?
> What else would you use to uniquely identify the arrays?  Not device
> names I hope.

Right now all that's in mdadm.conf is:

DEVICE /dev/sd[abcd]1
ARRAY /dev/md0 devices=/dev/sda1,/dev/sdb1,/dev/sdc1,/dev/sdd1
MAILADDR root@localhost

Something important missing..?

> > Should get me some sedatives for the day when this all explodes :P
>
> Just make sure it happens on your day off, then someone else will
> need the sedatives :-)

Yeah, the landlord.

-- 
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCS d--(+)@ s-:+ a- C+++(++++) UL+>++++ P+>++ L+++>++++ E-- W++ N o? K-
w--(---) !O M+ V- PS++(+) PE(-) Y++ PGP t++(---)@ 5 X+(++) R+(++) tv--(+)@ 
b++(+++) DI+++ D G++ e* h>++ r%>* y?
------END GEEK CODE BLOCK------

http://www.stop1984.com
http://www.againsttcpa.com

Re: softraid and multiple distros

[Neil Brown]

On Tuesday May 16, Dexter.Filmore@gmx.de wrote:
> 
> Right now all that's in mdadm.conf is:
> 
> DEVICE /dev/sd[abcd]1
> ARRAY /dev/md0 devices=/dev/sda1,/dev/sdb1,/dev/sdc1,/dev/sdd1
> MAILADDR root@localhost
> 
> Something important missing..?

If these are the only SCSI drives (or USB or anything else that looks
like a SCSI drive) that you have or will ever have, then it is
probably safe.
If you plug another device in and reboot, that other device could
conceivably become sda, and your current devices will be b,c,d,e.
If this happened, then mdadm could have trouble assembly your array.
At best it would be degraded as sde would not get included.

So in the very simple one-array situation, this is probably safe.
But it doesn't generalise.  If you have two array of distinct devices,
then something like 

 ARRAY /dev/md0 devices=/dev/sda1,/dev/sdb1,/dev/sdc1,/dev/sdd1
 ARRAY /dev/md1 devices=/dev/sde1,/dev/sdf1,/dev/sdg1,/dev/sdh1

is not safe.  If /dev/sda1 fails, you pull it out and reboot, then md1
won't be assembled properly as everything will get renamed.
In general it is safer to use UUIDs

  DEVICE /dev/sd[a-z][0-9]
  ARRAY /dev/md0 uuid=whatever
  ARRAY /dev/md1 uuid=whatever:else

Hope that makes it reasonably clear.

NeilBrown

Re: softraid and multiple distros

[Dexter Filmore]

> So in the very simple one-array situation, this is probably safe.
> But it doesn't generalise.  If you have two array of distinct devices,
> then something like
>
>  ARRAY /dev/md0 devices=/dev/sda1,/dev/sdb1,/dev/sdc1,/dev/sdd1
>  ARRAY /dev/md1 devices=/dev/sde1,/dev/sdf1,/dev/sdg1,/dev/sdh1
>
> is not safe.  If /dev/sda1 fails, you pull it out and reboot, then md1
> won't be assembled properly as everything will get renamed.
> In general it is safer to use UUIDs
>
>   DEVICE /dev/sd[a-z][0-9]
>   ARRAY /dev/md0 uuid=whatever
>   ARRAY /dev/md1 uuid=whatever:else
>
> Hope that makes it reasonably clear.

Absolutely, thanks.


-- 
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCS d--(+)@ s-:+ a- C+++(++++) UL+>++++ P+>++ L+++>++++ E-- W++ N o? K-
w--(---) !O M+ V- PS++(+) PE(-) Y++ PGP t++(---)@ 5 X+(++) R+(++) tv--(+)@ 
b++(+++) DI+++ D G++ e* h>++ r%>* y?
------END GEEK CODE BLOCK------

http://www.stop1984.com
http://www.againsttcpa.com

Re: softraid and multiple distros

[Neil Brown]

On Sunday May 14, Dexter.Filmore@gmx.de wrote:
> What do I need to do when I want to install a different distro on the machine 
> with a raid5 array?
> Which files do I need? /etc/mdadm.conf? /etc/raittab? both?

That depends a lot on the distros.

Some really old distro might be using /etc/raidtab, and some new
distros don't support it at all.
Some distros try to assemble all md arrays in the initrd, and some
mess this up in various ways.

If your root filesystem isn't on md/raid then it should be enough to
just preserve /etc/mdadm.conf, though it is fairly easy to recreate
that.  "mdadm -Escparititions" gives you a good starting point - edit
it to make it just right.

If your root is on md/raid, then you'll probably need to look into how
the initrd works (assuming the distros install program does just get
it right).

NeilBrown