[linux-lvm] Building up a RAID5 LVM home server (long)

[linux-lvm] Building up a RAID5 LVM home server (long)

[Erik Ohrnberger]

Dear Peeps of the LVM discussion list,

    In the past you've come to my rescue a number of times, and I wish to
thank you all for this assistance.

    I fear that my desires have once again out stripped my hands on
practical knowledge.  So I pose to you this discussion.  Mind you that if
there is a web page that covers the topic, please don't hesitate to point
into that direction.  This has been a self study project all along.

Some History: (skip if you want - see The Questions below) =============
    I've had a LVM up and running for some time for large file storage.  As
the storage needs grew, it was easy to add another hard disk, add it to the
volume group, and grow the file system.  This worked fairly seamlessly and
easily, and I figured it all out from the howtos and other information
resources.  The same held true for when the need for storage decreased and I
squeezed hard drives out of the file system and then the volume group.
Cool!  Up until one of the active hard disks with data died, and I lost
nearly all my data.  Oh well.  That's the way that it goes.  Thank the
computer gods that there was nothing of the data really all that terribly
irreplaceable, but still....

    After this I just went and got a 200 GB drive and left it at that (I was
going to school and had limited time for my computer addiction).  But now
that I'm done with that, I'm thinking of building a MythTV system, and I'm
certain that I will want to have a large amount of robust storage available
on the network.  So the question is what's the best way to build it?

The Questions:
==============
    It seems to me that RAID5 with at least one hot spare hard disk is one
of the safest ways to go for this type of storage.  The only concern that I
have is specific to the wide variety of hard disk sizes that I have
available (2 40GB, 1 60GB, 2 80GB, and I'll probably add the 200GB drive
once I've migrated that data off it to the array).  My limited understanding
of RAID5 is that it's best if all the hard drives are exactly the same.  Is
this true?  What are the downsides of using such a mix of hard disk sizes?

    Being able to resize the storage is a key, as is having a robust and
reliable storage pool.  As storage demands rise and fall, it's great to have
the flexibility to add and drop hard disks from the storage pool and use
them for other things, resizing the file system and the volume group as you
go along, of course.  If the storage pool is RAID5, and I add a larger hard
disk to the pool as a hot spare, and then use the software tools to fault
out the drive that I want, forcing a reconstruction, couldn't I pull the
faulted drive out, and use it for something else?  What sort of shape or
state will the RAID5 array be in at this point?  Will it use all of the
space on the newly added hot spare?

    Again, if there is a discussion thread that I've not found that covers
these questions and this topic, I will not be offended by a mere pointer to
the web page, I wish to educate myself about the trade offs to arrive at the
best possible compromise for my needs.

    Thanks as always and in advance.
        Erik.

Re: [linux-lvm] Building up a RAID5 LVM home server (long)

[Sam Vilain]

OK kid.  You've played quiet, done your homework and asked nicely, so
here's a recommended configuration for free.  Don't ask me how much I've
charged for this sort of thing distant times ago.

You've got yourself plenty of space.  So don't worry about RAID 5.  If you
ever find yourself thinking, "maybe I could make a little more space...",
forget it.  It's for certain niches that IMHO you don't fit into.
Especially with disks all different sizes like yours.  RAID 5 arrays with
hot spares are screaming madness, I tell you.

Look at it this way: with RAID 1, just about all access to the filesystem
is at worst the same speed as before (sans a little extra load on the
system sending the data to the IDE controller twice), or twice as fast
(for reads).  For any application with heavy random access patterns this
makes a considerable difference.  Also, it means streaming applications
don't kill access times.  All this, for one more disk than a 5 disk RAID5
with hot spare array.

You've got 2x40GB.  Those will be good as base disks for a system.

Normally it's best for performance to pair up identical disks throughout
a system for mirrors, do I'd be inclined to go for something like this:

   40GB - /dev/hda
    /dev/hda1 -  1GB - root (/dev/md0)
    /dev/hda2 -  1GB - swap (/dev/md1)
    /dev/hda3 - 38GB - LVM space (PV: /dev/md2)

   80GB - /dev/hdb
     whole disk - LVM space (PV: /dev/md3)

   40GB - /dev/hdc
   mirror pair of /dev/hda
    /dev/hdc1 -  1GB - root (/dev/md0)
    /dev/hdc2 -  1GB - swap (/dev/md1)
    /dev/hdc3 - 38GB - LVM space (PV: /dev/md2)

   80GB - /dev/hdd
   mirror pair of /dev/hdb
     whole disk - LVM space (PV: /dev/md3)

  200GB - /dev/hde
    /dev/hde1 - 60GB - LVM space (PV: /dev/md4)
    /dev/hde2 - 140GB - LVM space (PV: /dev/md5)

   60GB - /dev/hdg
   mirror pair of first partition on /dev/hde
     whole disk - LVM space (PV: /dev/md4)

In this setup, you end up with:

   - two mirrored "raw" partitions, that you can use for a root filesystem,
     and swap.  Or turn swap off while you install a different distribution
     into one of them for testing.  You'll at some point want to move your
     /usr to an LVM partition, which means that you'll want to keep an LVM
     `cheat sheet' on hand in case your LVM doesn't come up and you need to
     use the admin utilities.  Either that, or only put /usr/local on LVM,
     keeping /usr minimal.  Or just make a bigger / :).
   - total 38GB + 60GB + 80GB = 180GB *fast* mirrored LVM space
   - total 140GB *fast* unmirrored space

the unmirrored space would be great for keeping large caches of stuff
you already have on media.  everything else, the OS and things that you
want random access to be as fast as possible.

You could also then use `pvmove' to move stuff from mirrored to unmirrored
space on the LVM.

Even though your unmirrored space has no pair currently, notice how it's
still got a mirror device in the above plan - /dev/md5.  This will be set
up as a "degenerate" (one-way) mirror, using an /etc/raidtab entry like:

  raiddev /dev/md5
    raid-level       1
    nr-raid-disks    2
    nr-spare-disks   0
    persistent-superblock 1
    chunk-size        4
    device   /dev/hde2
    raid-disk 0
    device   /dev/hdg2
    raid-disk 1
    failed-disk 1

This will allow you to attach a mirror pair to that partition later
without much effort at all.

In fact, in practice setting a whole system up like this in the first
place is a great way to prototype a layout without losing the ability to
back-out (sans inconveniently timed disasters).  Then you can set up the
partition tables when you're happy, `raidhotadd' to attach the other half
of the mirrors, and `cat /proc/mdstat' to kill time while it syncs.

...

Alternatives exist, which break the "mirror identical disks across
controllers" best mirroring practice, but maximise mirrored space:

   40GB - /dev/hda
    /dev/hda1 -  1GB - root (/dev/md0)
    /dev/hda2 -  1GB - swap (/dev/md1)
    /dev/hda3 - 38GB - LVM space (PV: /dev/md2)

   40GB - /dev/hdc
   mirror pair of /dev/hda
    /dev/hdc1 -  1GB - root (/dev/md0)
    /dev/hdc2 -  1GB - swap (/dev/md1)
    /dev/hdc3 - 38GB - LVM space (PV: /dev/md2)

   60GB - /dev/hdd
   mirror pair of last /dev/hde partition
    /dev/hdd1 - 40GB - LVM space (PV: /dev/md5)
    /dev/hdd2 - 20GB - unmirrored LVM space (PV: /dev/md6)

  200GB - /dev/hde
    /dev/hde1 - 80GB - LVM space (PV: /dev/md3)
    /dev/hde2 - 80GB - LVM space (PV: /dev/md4)
    /dev/hde3 - 40GB - LVM space (PV: /dev/md5)

   80GB - /dev/hdg
     whole disk - LVM space (PV: /dev/md3)

   80GB - /dev/hdh
   mirror pair of /dev/hdb
     whole disk - LVM space (PV: /dev/md4)

This gives you:

   - total 38GB + 80GB + 80GB + 40GB = 238GB *fast* mirrored LVM space
   - total 20GB unmirrored space

...

Hopefully that should give you enough of a hint about how best to go
about arranging these sorts of things :)

The underlying principle, is keeping the MD layer as simple as possible,
so that you've got nice little resilient building blocks for your Volume
Group.  Never use concatenation, striping or RAID5 at the MD layer if
you've got LVM, it's simply in the wrong place!

You'll need to do a little work to get GRUB to be able to boot off the
mirrored root.  But it's well worth it.  "kernel RAID device
autodetection" is a highly recommended kernel option.

HTH,
Sam.



Erik Ohrnberger wrote:
> Dear Peeps of the LVM discussion list,
> 
>     In the past you've come to my rescue a number of times, and I wish to
> thank you all for this assistance.
> 
>     I fear that my desires have once again out stripped my hands on
> practical knowledge.  So I pose to you this discussion.  Mind you that if
> there is a web page that covers the topic, please don't hesitate to point
> into that direction.  This has been a self study project all along.
> 
> Some History: (skip if you want - see The Questions below) =============
>     I've had a LVM up and running for some time for large file storage.  As
> the storage needs grew, it was easy to add another hard disk, add it to the
> volume group, and grow the file system.  This worked fairly seamlessly and
> easily, and I figured it all out from the howtos and other information
> resources.  The same held true for when the need for storage decreased and I
> squeezed hard drives out of the file system and then the volume group.
> Cool!  Up until one of the active hard disks with data died, and I lost
> nearly all my data.  Oh well.  That's the way that it goes.  Thank the
> computer gods that there was nothing of the data really all that terribly
> irreplaceable, but still....
> 
>     After this I just went and got a 200 GB drive and left it at that (I was
> going to school and had limited time for my computer addiction).  But now
> that I'm done with that, I'm thinking of building a MythTV system, and I'm
> certain that I will want to have a large amount of robust storage available
> on the network.  So the question is what's the best way to build it?
> 
> The Questions:
> ==============
>     It seems to me that RAID5 with at least one hot spare hard disk is one
> of the safest ways to go for this type of storage.  The only concern that I
> have is specific to the wide variety of hard disk sizes that I have
> available (2 40GB, 1 60GB, 2 80GB, and I'll probably add the 200GB drive
> once I've migrated that data off it to the array).  My limited understanding
> of RAID5 is that it's best if all the hard drives are exactly the same.  Is
> this true?  What are the downsides of using such a mix of hard disk sizes?
> 
>     Being able to resize the storage is a key, as is having a robust and
> reliable storage pool.  As storage demands rise and fall, it's great to have
> the flexibility to add and drop hard disks from the storage pool and use
> them for other things, resizing the file system and the volume group as you
> go along, of course.  If the storage pool is RAID5, and I add a larger hard
> disk to the pool as a hot spare, and then use the software tools to fault
> out the drive that I want, forcing a reconstruction, couldn't I pull the
> faulted drive out, and use it for something else?  What sort of shape or
> state will the RAID5 array be in at this point?  Will it use all of the
> space on the newly added hot spare?
> 
>     Again, if there is a discussion thread that I've not found that covers
> these questions and this topic, I will not be offended by a mere pointer to
> the web page, I wish to educate myself about the trade offs to arrive at the
> best possible compromise for my needs.
> 
>     Thanks as always and in advance.
>         Erik.
> 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> 


-- 
Sam Vilain, sam /\T vilain |><>T net, PGP key ID: 0x05B52F13
(include my PGP key ID in personal replies to avoid spam filtering)

RE: [linux-lvm] lvm liveCD help needed

[Bao, Liping]

 
Hi Paul

Several things:
1.several method to mount your root
	a)using "raidautorun" to detect the MDs.
	b)using "raidstart -a -c $RAIDTAB",
	 here you may can not get the raidtab file, you need to compile
one manually. 
	Alternatilvey, "mount /dev/hda3 /mnt/disks", (because the / is
md1, and I assume that /dev/hda3 is one of the partition in the md1).

2./boot can be installed on md0?


Message: 4
Date: Wed, 23 Feb 2005 16:43:41 -0500
From: Paul Pianta <pantz@lqt.ca>
Subject: [linux-lvm] lvm liveCD help needed
To: linux-lvm@redhat.com
Message-ID: <1109195021.4695.65.camel@aragorn.intranet>
Content-Type: text/plain

Hi people

I have a little problem that I was hoping someone might be able to help
me with.

I 'text mode' installed a FC1 installation and was unable to configure
the volume groups or logical volumes (limitation of text mode :( ...). I
thought "that's ok - I will just create them later!" - and this is where
my problem began :)

I configured the 2 x 200G ide drives with software raid as follows:
/dev/md0 - /boot - 100M
/dev/md1 - /     - 10G
/dev/md2 - swap
/dev/md3 - one big physical volume (lvm) ~185G

The system is installed and I assume that it all went onto /dev/md1.
What I wanted to do was to put /usr, /var, and /home in logical volumes
in one large volume group as such:

/vgRoot/lvUsr - 10G
/vgRoot/lvHome - 20G
/vgRoot/lvVar - 120G
with 30G free for playing around with later

Now ...

I have a gentoo live cd (with lvm support) and I assume this is what I
need to use to 
1. create the volume group vgRoot
2. create the three logical volumes lvUsr, lvHome, lvVar
3. copy /usr, /home, /var from /dev/md1 over to these newly created
logical volumes
4. edit /etc/fstab to reflect the new partition layout
5. reboot and dance around the room with my pants down singing 'oh what
a beautiful morning'


My problem is I don't know how to access the lvm support on the gentoo
live cd and even worse than that - I don't know how to get at my /
partition with the live cd because it is a software raid partition and I
don't know how to go about mounting it.

Any ideas would be much appreciated - and I will promise not to drop my
pants whilst singing if we can get this figured out :)

thanks

pantz

[linux-lvm] Building up a RAID5 LVM home server (long)

[Erik Ohrnberger]

Dear Peeps of the LVM discussion list,

    In the past you've come to my rescue a number of times, and I wish to
thank you all for this assistance.

    I fear that my desires have once again out stripped my hands on
practical knowledge.  So I pose to you this discussion.  Mind you that if
there is a web page that covers the topic, please don't hesitate to point
into that direction.  This has been a self study project all along.

Some History: (skip if you want - see The Questions below)
=============
    I've had a LVM up and running for some time for large file storage.  As
the storage needs grew, it was easy to add another hard disk, add it to the
volume group, and grow the file system.  This worked fairly seamlessly and
easily, and I figured it all out from the howtos and other information
resources.  The same held true for when the need for storage decreased and I
squeezed hard drives out of the file system and then the volume group.
Cool!  Up until one of the active hard disks with data died, and I lost
nearly all my data.  Oh well.  That's the way that it goes.  Thank the
computer gods that there was nothing of the data really all that terribly
irreplaceable, but still....

    After this I just went and got a 200 GB drive and left it at that (I was
going to school and had limited time for my computer addiction).  But now
that I'm done with that, I'm thinking of building a MythTV system, and I'm
certain that I will want to have a large amount of robust storage available
on the network.  So the question is what's the best way to build it?

The Questions:
==============
    It seems to me that RAID5 with at least one hot spare hard disk is one
of the safest ways to go for this type of storage.  The only concern that I
have is specific to the wide variety of hard disk sizes that I have
available (2 40GB, 1 60GB, 2 80GB, and I'll probably add the 200GB drive
once I've migrated that data off it to the array).  My limited understanding
of RAID5 is that it's best if all the hard drives are exactly the same.  Is
this true?  What are the downsides of using such a mix of hard disk sizes?

    Being able to resize the storage is a key, as is having a robust and
reliable storage pool.  As storage demands rise and fall, it's great to have
the flexibility to add and drop hard disks from the storage pool and use
them for other things, resizing the file system and the volume group as you
go along, of course.  If the storage pool is RAID5, and I add a larger hard
disk to the pool as a hot spare, and then use the software tools to fault
out the drive that I want, forcing a reconstruction, couldn't I pull the
faulted drive out, and use it for something else?  What sort of shape or
state will the RAID5 array be in at this point?  Will it use all of the
space on the newly added hot spare?

    Again, if there is a discussion thread that I've not found that covers
these questions and this topic, I will not be offended by a mere pointer to
the web page, I wish to educate myself about the trade offs to arrive at the
best possible compromise for my needs.

    Thanks as always and in advance.
        Erik.

Re: [linux-lvm] Building up a RAID5 LVM home server (long)

[Scott Serr]

Erik Ohrnberger wrote:

>Dear Peeps of the LVM discussion list,
>
>    In the past you've come to my rescue a number of times, and I wish to
>thank you all for this assistance.
>
>    I fear that my desires have once again out stripped my hands on
>practical knowledge.  So I pose to you this discussion.  Mind you that if
>there is a web page that covers the topic, please don't hesitate to point
>into that direction.  This has been a self study project all along.
>
>Some History: (skip if you want - see The Questions below)
>=============
>    I've had a LVM up and running for some time for large file storage.  As
>the storage needs grew, it was easy to add another hard disk, add it to the
>volume group, and grow the file system.  This worked fairly seamlessly and
>easily, and I figured it all out from the howtos and other information
>resources.  The same held true for when the need for storage decreased and I
>squeezed hard drives out of the file system and then the volume group.
>Cool!  Up until one of the active hard disks with data died, and I lost
>nearly all my data.  Oh well.  That's the way that it goes.  Thank the
>computer gods that there was nothing of the data really all that terribly
>irreplaceable, but still....
>
>    After this I just went and got a 200 GB drive and left it at that (I was
>going to school and had limited time for my computer addiction).  But now
>that I'm done with that, I'm thinking of building a MythTV system, and I'm
>certain that I will want to have a large amount of robust storage available
>on the network.  So the question is what's the best way to build it?
>
>The Questions:
>==============
>    It seems to me that RAID5 with at least one hot spare hard disk is one
>of the safest ways to go for this type of storage.  The only concern that I
>have is specific to the wide variety of hard disk sizes that I have
>available (2 40GB, 1 60GB, 2 80GB, and I'll probably add the 200GB drive
>once I've migrated that data off it to the array).  My limited understanding
>of RAID5 is that it's best if all the hard drives are exactly the same.  Is
>this true?  What are the downsides of using such a mix of hard disk sizes?
>  
>
The down side is the partitions that make up a RAID5 have to match in 
size, if they don't - the RAID5 just uses the minimum partition size of 
the set for EACH partition.  So if you have 20GB, 30GB, 40GB.  10GB of 
the 30GB will be wasted.  20GB of the 40GB will be wasted.  So you might 
as well use the wasted space for scratch etc.  You can optimize your 
disk use but you never want to include TWO partitions from one disk in 
the same RAID set.  Right?

>    Being able to resize the storage is a key, as is having a robust and
>reliable storage pool.  As storage demands rise and fall, it's great to have
>the flexibility to add and drop hard disks from the storage pool and use
>them for other things, resizing the file system and the volume group as you
>go along, of course.  If the storage pool is RAID5, and I add a larger hard
>disk to the pool as a hot spare, and then use the software tools to fault
>out the drive that I want, forcing a reconstruction, couldn't I pull the
>faulted drive out, and use it for something else?  What sort of shape or
>state will the RAID5 array be in at this point?  Will it use all of the
>space on the newly added hot spare?
>  
>
I haven't use hot spares on Linux, a little on Solaris.  You could do 
what you say in theory.  But normally on low budget stuff it's not "hot 
plug" so you would have to shutdown and pull out your main drive.  In my 
situation this would be bad, because I don't do my / (root) on RAID5.  I 
could boot my "backup" root that I make with rsync, but then I would 
have to fix the fstab and make sure GRUB is installed on there and have 
a BIOS that will point to hdb (not just hda) for booting. 

>    Again, if there is a discussion thread that I've not found that covers
>these questions and this topic, I will not be offended by a mere pointer to
>the web page, I wish to educate myself about the trade offs to arrive at the
>best possible compromise for my needs.
>
>    Thanks as always and in advance.
>        Erik.
>
>
>_______________________________________________
>linux-lvm mailing list
>linux-lvm@redhat.com
>https://www.redhat.com/mailman/listinfo/linux-lvm
>read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>  
>

Re: [linux-lvm] Building up a RAID5 LVM home server (long)

[Erik Ohrnberger]

On Tue, March 1, 2005 11:43, Scott Serr said:
> Erik Ohrnberger wrote:
>
>>Dear Peeps of the LVM discussion list,

....snip....

>>The Questions:
>>==============
>>    It seems to me that RAID5 with at least one hot spare hard disk is
>> one
>>of the safest ways to go for this type of storage.  The only concern that
>> I
>>have is specific to the wide variety of hard disk sizes that I have
>>available (2 40GB, 1 60GB, 2 80GB, and I'll probably add the 200GB drive
>>once I've migrated that data off it to the array).  My limited
>> understanding
>>of RAID5 is that it's best if all the hard drives are exactly the same.
>> Is
>>this true?  What are the downsides of using such a mix of hard disk
>> sizes?
>>
>>
> The down side is the partitions that make up a RAID5 have to match in
> size, if they don't - the RAID5 just uses the minimum partition size of
> the set for EACH partition.  So if you have 20GB, 30GB, 40GB.  10GB of
> the 30GB will be wasted.  20GB of the 40GB will be wasted.  So you might
> as well use the wasted space for scratch etc.  You can optimize your
> disk use but you never want to include TWO partitions from one disk in
> the same RAID set.  Right?

Not including two partitions from the same drive into the same raid set
would make sense.  What would the redundancy of that be?  It wouldn't be.

However, I've been thinking on this.  What if I took two small drives
and raid-0'd them into a single larger block device, and then included
that into the raid5 set.  Is this possible?

20GB + 10GB (of 40GB) = md0 (raid 0)  then
md0 + 40GB (what's left) + 40 GB = md1 (raid 5)

Umm, right.  That would break the rules, 40GB is contributing twice to the
raid5 set.  Hmmm.

What if the I broke everything into 10 GB pieces, and created multiple raid5
sets?  Then I could LVM2 them together and have a large filesystem that way.

a=20GB, b=30GB, c=40GB

a-1 + b-1 + c-1 = md0 (approx 30 GB storage)
a-2 + b-2 + c-2 = md1 (approx 30 GB storage)
      b-3 + c-3 = md2 (waiting for one more drive)
            c-4 = md3 (waiting for two more drives)


>>    Being able to resize the storage is a key, as is having a robust and
>>reliable storage pool.  As storage demands rise and fall, it's great to
>> have
>>the flexibility to add and drop hard disks from the storage pool and use
>>them for other things, resizing the file system and the volume group as
>> you
>>go along, of course.  If the storage pool is RAID5, and I add a larger
>> hard
>>disk to the pool as a hot spare, and then use the software tools to fault
>>out the drive that I want, forcing a reconstruction, couldn't I pull the
>>faulted drive out, and use it for something else?  What sort of shape or
>>state will the RAID5 array be in at this point?  Will it use all of the
>>space on the newly added hot spare?
>>
>>
> I haven't use hot spares on Linux, a little on Solaris.  You could do
> what you say in theory.  But normally on low budget stuff it's not "hot
> plug" so you would have to shutdown and pull out your main drive.  In my
> situation this would be bad, because I don't do my / (root) on RAID5.  I
> could boot my "backup" root that I make with rsync, but then I would
> have to fix the fstab and make sure GRUB is installed on there and have
> a BIOS that will point to hdb (not just hda) for booting.

Well, I wasn't thinking hot swapping.  I'd shut the machine down to add and
remove hard disks, but the idea is to make use of the reconstruction as a
means for migrating hard disks into and out of the raid array, and it sounds
like that would work OK.

... snip ...

Re: [linux-lvm] Building up a RAID5 LVM home server (long)

[Scott Serr]

Erik Ohrnberger wrote:

>On Tue, March 1, 2005 11:43, Scott Serr said:
>  
>
>>Erik Ohrnberger wrote:
>>
>>    
>>
>>>Dear Peeps of the LVM discussion list,
>>>      
>>>
>
>....snip....
>
>  
>
>>>The Questions:
>>>==============
>>>   It seems to me that RAID5 with at least one hot spare hard disk is
>>>one
>>>of the safest ways to go for this type of storage.  The only concern that
>>>I
>>>have is specific to the wide variety of hard disk sizes that I have
>>>available (2 40GB, 1 60GB, 2 80GB, and I'll probably add the 200GB drive
>>>once I've migrated that data off it to the array).  My limited
>>>understanding
>>>of RAID5 is that it's best if all the hard drives are exactly the same.
>>>Is
>>>this true?  What are the downsides of using such a mix of hard disk
>>>sizes?
>>>
>>>
>>>      
>>>
>>The down side is the partitions that make up a RAID5 have to match in
>>size, if they don't - the RAID5 just uses the minimum partition size of
>>the set for EACH partition.  So if you have 20GB, 30GB, 40GB.  10GB of
>>the 30GB will be wasted.  20GB of the 40GB will be wasted.  So you might
>>as well use the wasted space for scratch etc.  You can optimize your
>>disk use but you never want to include TWO partitions from one disk in
>>the same RAID set.  Right?
>>    
>>
>
>Not including two partitions from the same drive into the same raid set
>would make sense.  What would the redundancy of that be?  It wouldn't be.
>
>However, I've been thinking on this.  What if I took two small drives
>and raid-0'd them into a single larger block device, and then included
>that into the raid5 set.  Is this possible?
>
>20GB + 10GB (of 40GB) = md0 (raid 0)  then
>md0 + 40GB (what's left) + 40 GB = md1 (raid 5)
>
>Umm, right.  That would break the rules, 40GB is contributing twice to the
>raid5 set.  Hmmm.
>
>What if the I broke everything into 10 GB pieces, and created multiple raid5
>sets?  Then I could LVM2 them together and have a large filesystem that way.
>
>a=20GB, b=30GB, c=40GB
>
>a-1 + b-1 + c-1 = md0 (approx 30 GB storage)
>a-2 + b-2 + c-2 = md1 (approx 30 GB storage)
>      b-3 + c-3 = md2 (waiting for one more drive)
>            c-4 = md3 (waiting for two more drives)
>  
>
This is sorta what I do.  But in my opinion the gain of having RAID5 
(over RAID1) is when you get over 3 disks...  at 3 disks you are burning 
33% for redudnacy... 25% or 20% or  17% sounds better to me.  I guess if 
you go too far it costs in calculating the parity.

>>>   Being able to resize the storage is a key, as is having a robust and
>>>reliable storage pool.  As storage demands rise and fall, it's great to
>>>have
>>>the flexibility to add and drop hard disks from the storage pool and use
>>>them for other things, resizing the file system and the volume group as
>>>you
>>>go along, of course.  If the storage pool is RAID5, and I add a larger
>>>hard
>>>disk to the pool as a hot spare, and then use the software tools to fault
>>>out the drive that I want, forcing a reconstruction, couldn't I pull the
>>>faulted drive out, and use it for something else?  What sort of shape or
>>>state will the RAID5 array be in at this point?  Will it use all of the
>>>space on the newly added hot spare?
>>>
>>>
>>>      
>>>
>>I haven't use hot spares on Linux, a little on Solaris.  You could do
>>what you say in theory.  But normally on low budget stuff it's not "hot
>>plug" so you would have to shutdown and pull out your main drive.  In my
>>situation this would be bad, because I don't do my / (root) on RAID5.  I
>>could boot my "backup" root that I make with rsync, but then I would
>>have to fix the fstab and make sure GRUB is installed on there and have
>>a BIOS that will point to hdb (not just hda) for booting.
>>    
>>
>
>Well, I wasn't thinking hot swapping.  I'd shut the machine down to add and
>remove hard disks, but the idea is to make use of the reconstruction as a
>means for migrating hard disks into and out of the raid array, and it sounds
>like that would work OK.
>  
>
I've thought about it...  But once you have close to a terrabyte of 
stuff that isn't backed up, well...  atleast I whimped out on this.  I'm 
sure it would work in theory and 99% in practice.

RE: [linux-lvm] Building up a RAID5 LVM home server (long)

[Erik Ohrnberger]

... SNIP ...
> >What if the I broke everything into 10 GB pieces, and 
> created multiple 
> >raid5 sets?  Then I could LVM2 them together and have a large 
> >filesystem that way.
> >
> >a=20GB, b=30GB, c=40GB
> >
> >a-1 + b-1 + c-1 = md0 (approx 30 GB storage)
> >a-2 + b-2 + c-2 = md1 (approx 30 GB storage)
> >      b-3 + c-3 = md2 (waiting for one more drive)
> >            c-4 = md3 (waiting for two more drives)
> >  
> >
> This is sorta what I do.  But in my opinion the gain of having RAID5 
> (over RAID1) is when you get over 3 disks...  at 3 disks you 
> are burning 
> 33% for redudnacy... 25% or 20% or  17% sounds better to me.  
> I guess if 
> you go too far it costs in calculating the parity.

Overhead: Yea, OK. Nothing is without a price.
I fooled around with various ideas, and came up with this for my
particulars:
(Note, rounded to nearest GB)

80 GB	/dev/hda	60 GB /dev/hdb	40 GB	/dev/hdc	45 GB
/dev/hdd

GB		/dev/md0 (RAID0)
40		/dev/hdc
15		/dev/hda1
55		
		
		/dev/md1 (RAID0)
45		/dev/hdd
10		/dev/hda2
55		
		
		/dev/md2 (RAID5)
55		/dev/md0
55		/dev/md1
55		/dev/hda3
55		/dev/hdb
220

Yea, OK, so like the 220 is a bit optimistic, but should get pretty close to
that.

What do you think?

Re: [linux-lvm] Building up a RAID5 LVM home server (long)

[Ron Watkins]

This is absolutely insane.   It is among the dumbest designs I've ever seen. 
You are using HDA as a component in MD0 and MD1, and then using HDA AGAIN as 
part of MD2 directly, while using it indirectly via MD0 and MD1.   You're 
going to A) bottleneck on HDA, B) you're going to beat that drive to death, 
and C) if that drive goes, you are HOSED.  Plus, you are just begging for 
problems with potential bugs in the RAID driver code.  This whole setup is 
an INCREDIBLY bad idea.  You're trying to 'be clever' to save yourself some 
money, and all you're doing is buying trouble.

The way RAID5 is meant to work is with disks of approximately the same size. 
RAID5 is not expandable, unless you have a very expensive hardware 
controller.  There are algorithms that will let you expand the size of a 
RAID5 volume, but they have not, to my knowledge, been implemented in open 
source.   You CANNOT do what you want to do, cheaply.  You can spend a great 
deal of money to satisfy most of your design parameters, but NOT cheaply. 
If you want it cheap, use fixed drives of about the same size, and don't 
think about expansion.  When you're ready to expand, hook up another, bigger 
RAID and copy your data.   In NO case can you use that hodgepodge of junk 
drives you've collected.

Most of your drives are obsolete.  Keep the biggest one, buy at least two 
more of the same size, and set up a RAID5 using that.  All this monkeying 
around to try to extract some last value from drives totally ill-suited for 
the purpose is going to cost you far, far more than new drives ever could.

Hell, keep the smaller ones around, put them into a concatenated LVM2 
volume, and use them as a backup.  It's not the best backup in the world, 
but it's better than nothing.

Do it right.  This is your data you're trying to save.  You can get very 
nice 250-gig PATA Western Digital drives for $165 from www.newegg.com.  They 
are specifically designed for RAID.  Buy 4 and save yourself this massive 
headache.  If you don't need that much space, buy smaller drives.

Or, you can persist in trying to be clever, but it's a virtual *certainty* 
you're going to lose data if you go this route.  Pay now, or pay later. 
What's your data worth?

<<RON>>

----- Original Message ----- 
From: "Erik Ohrnberger" <erik@echohome.org>
To: "'LVM general discussion and development'" <linux-lvm@redhat.com>
Sent: Tuesday, March 01, 2005 10:27 PM
Subject: RE: [linux-lvm] Building up a RAID5 LVM home server (long)


>
> ... SNIP ...
>> >What if the I broke everything into 10 GB pieces, and
>> created multiple
>> >raid5 sets?  Then I could LVM2 them together and have a large
>> >filesystem that way.
>> >
>> >a=20GB, b=30GB, c=40GB
>> >
>> >a-1 + b-1 + c-1 = md0 (approx 30 GB storage)
>> >a-2 + b-2 + c-2 = md1 (approx 30 GB storage)
>> >      b-3 + c-3 = md2 (waiting for one more drive)
>> >            c-4 = md3 (waiting for two more drives)
>> >
>> >
>> This is sorta what I do.  But in my opinion the gain of having RAID5
>> (over RAID1) is when you get over 3 disks...  at 3 disks you
>> are burning
>> 33% for redudnacy... 25% or 20% or  17% sounds better to me.
>> I guess if
>> you go too far it costs in calculating the parity.
>
> Overhead: Yea, OK. Nothing is without a price.
> I fooled around with various ideas, and came up with this for my
> particulars:
> (Note, rounded to nearest GB)
>
> 80 GB /dev/hda 60 GB /dev/hdb 40 GB /dev/hdc 45 GB
> /dev/hdd
>
> GB /dev/md0 (RAID0)
> 40 /dev/hdc
> 15 /dev/hda1
> 55
>
> /dev/md1 (RAID0)
> 45 /dev/hdd
> 10 /dev/hda2
> 55
>
> /dev/md2 (RAID5)
> 55 /dev/md0
> 55 /dev/md1
> 55 /dev/hda3
> 55 /dev/hdb
> 220
>
> Yea, OK, so like the 220 is a bit optimistic, but should get pretty close 
> to
> that.
>
> What do you think?
>
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> 

RE: [linux-lvm] Building up a RAID5 LVM home server (long)

[Erik Ohrnberger]

Ron,
    Well, there is no mistaking this feedback  ;-)

    Thanks for the direct, succinct feedback.  I think you are right in your
points.

    The search goes on to balance flexibility, cost, and benefits, as
always.

    Erik.

> -----Original Message-----
> From: linux-lvm-bounces@redhat.com 
> [mailto:linux-lvm-bounces@redhat.com] On Behalf Of Ron Watkins
> Sent: Tuesday, March 01, 2005 11:10 PM
> To: Erik@echohome.org; LVM general discussion and development
> Subject: Re: [linux-lvm] Building up a RAID5 LVM home server (long)
> 
> 
> This is absolutely insane.   It is among the dumbest designs 
> I've ever seen. 
> You are using HDA as a component in MD0 and MD1, and then 
> using HDA AGAIN as 
> part of MD2 directly, while using it indirectly via MD0 and 
> MD1.   You're 
> going to A) bottleneck on HDA, B) you're going to beat that 
> drive to death, 
> and C) if that drive goes, you are HOSED.  Plus, you are just 
> begging for 
> problems with potential bugs in the RAID driver code.  This 
> whole setup is 
> an INCREDIBLY bad idea.  You're trying to 'be clever' to save 
> yourself some 
> money, and all you're doing is buying trouble.
> 
> The way RAID5 is meant to work is with disks of approximately 
> the same size. 
> RAID5 is not expandable, unless you have a very expensive hardware 
> controller.  There are algorithms that will let you expand 
> the size of a 
> RAID5 volume, but they have not, to my knowledge, been 
> implemented in open 
> source.   You CANNOT do what you want to do, cheaply.  You 
> can spend a great 
> deal of money to satisfy most of your design parameters, but 
> NOT cheaply. 
> If you want it cheap, use fixed drives of about the same 
> size, and don't 
> think about expansion.  When you're ready to expand, hook up 
> another, bigger 
> RAID and copy your data.   In NO case can you use that 
> hodgepodge of junk 
> drives you've collected.
> 
> Most of your drives are obsolete.  Keep the biggest one, buy 
> at least two 
> more of the same size, and set up a RAID5 using that.  All 
> this monkeying 
> around to try to extract some last value from drives totally 
> ill-suited for 
> the purpose is going to cost you far, far more than new 
> drives ever could.
> 
> Hell, keep the smaller ones around, put them into a concatenated LVM2 
> volume, and use them as a backup.  It's not the best backup 
> in the world, 
> but it's better than nothing.
> 
> Do it right.  This is your data you're trying to save.  You 
> can get very 
> nice 250-gig PATA Western Digital drives for $165 from 
> www.newegg.com.  They 
> are specifically designed for RAID.  Buy 4 and save yourself 
> this massive 
> headache.  If you don't need that much space, buy smaller drives.
> 
> Or, you can persist in trying to be clever, but it's a 
> virtual *certainty* 
> you're going to lose data if you go this route.  Pay now, or 
> pay later. 
> What's your data worth?
> 
> <<RON>>
> 
> ----- Original Message ----- 
> From: "Erik Ohrnberger" <erik@echohome.org>
> To: "'LVM general discussion and development'" <linux-lvm@redhat.com>
> Sent: Tuesday, March 01, 2005 10:27 PM
> Subject: RE: [linux-lvm] Building up a RAID5 LVM home server (long)
> 
> 
> >
> > ... SNIP ...
> >> >What if the I broke everything into 10 GB pieces, and
> >> created multiple
> >> >raid5 sets?  Then I could LVM2 them together and have a large 
> >> >filesystem that way.
> >> >
> >> >a=20GB, b=30GB, c=40GB
> >> >
> >> >a-1 + b-1 + c-1 = md0 (approx 30 GB storage)
> >> >a-2 + b-2 + c-2 = md1 (approx 30 GB storage)
> >> >      b-3 + c-3 = md2 (waiting for one more drive)
> >> >            c-4 = md3 (waiting for two more drives)
> >> >
> >> >
> >> This is sorta what I do.  But in my opinion the gain of 
> having RAID5 
> >> (over RAID1) is when you get over 3 disks...  at 3 disks you are 
> >> burning 33% for redudnacy... 25% or 20% or  17% sounds 
> better to me.
> >> I guess if
> >> you go too far it costs in calculating the parity.
> >
> > Overhead: Yea, OK. Nothing is without a price.
> > I fooled around with various ideas, and came up with this for my
> > particulars:
> > (Note, rounded to nearest GB)
> >
> > 80 GB /dev/hda 60 GB /dev/hdb 40 GB /dev/hdc 45 GB
> > /dev/hdd
> >
> > GB /dev/md0 (RAID0)
> > 40 /dev/hdc
> > 15 /dev/hda1
> > 55
> >
> > /dev/md1 (RAID0)
> > 45 /dev/hdd
> > 10 /dev/hda2
> > 55
> >
> > /dev/md2 (RAID5)
> > 55 /dev/md0
> > 55 /dev/md1
> > 55 /dev/hda3
> > 55 /dev/hdb
> > 220
> >
> > Yea, OK, so like the 220 is a bit optimistic, but should get pretty 
> > close
> > to
> > that.
> >
> > What do you think?
> >
> >
> > _______________________________________________
> > linux-lvm mailing list
> > linux-lvm@redhat.com 
> https://www.redhat.com/mailman/listinfo/linux-> lvm
> > read the 
> LVM HOW-TO at 
> http://tldp.org/HOWTO/LVM-HOWTO/
> > 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> 

Re: [linux-lvm] Building up a RAID5 LVM home server (long)

[Ron Watkins]

Unfortunately, RAID5 just isn't very flexible.   In exchange, you get to use 
most of your purchased drive space, you don't have to spend a mountain of 
money, and you get pretty reasonable speed.  (slow on writes, quite fast on 
reads).  RAID-10 is faster, and can often take more drives failing (small 
chance of up to half), but costs you 50% of your storage space.

It's a bit suboptimal, but you can, using LVM, expand a RAID-5, by adding 
more discrete RAID-5 PVs.  Each individual RAID array should be made from 
similar drives, but you can stripe across multiple physical RAIDs that way. 
This is, however, less reliable.  Whenever you stripe across multiple 
volumes, if any of the volumes fail, you lose everything.  RAID5 volumes are 
less likely to fail, but whatever that risk is, you're taking multiples of 
that risk by striping across multiple volumes.

I'm sorry I was so annoyed-sounding, I don't normally flame too much.  That 
last question I asked was actually somewhat serious... what is your data 
WORTH?  If it's just stuff you can easily recreate, like video files off 
Usenet, then you may not need a RAID at all.  Just join all your existing 
disks together.  You have a high chance of failure that way, but if you 
don't care about the data, big deal.  A failure just costs you some 
rebuilding time.

If you do care about the data, and it sounds like you do, think about how 
much it would cost you to replace, and budget accordingly.  My pointed words 
were coming from a background of 'data loss is a catastrophe!'.  If it's not 
unique/irreplaceable, maybe just burning it to CD every once in awhile would 
be enough.

If you really do want the data protection of a RAID, your design won't get 
you there. Do it right... spend enough money to do a good job.  Otherwise, 
just back up stuff you care about.

You can often save money by being clever, but RAID itself is already 
exceedingly clever.  No matter how much thinking time you do, you're not 
likely to improve on it much.

----- Original Message ----- 
From: "Erik Ohrnberger" <erik@echohome.org>
To: "'LVM general discussion and development'" <linux-lvm@redhat.com>
Sent: Tuesday, March 01, 2005 11:27 PM
Subject: RE: [linux-lvm] Building up a RAID5 LVM home server (long)


> Ron,
>    Well, there is no mistaking this feedback  ;-)
>
>    Thanks for the direct, succinct feedback.  I think you are right in 
> your
> points.
>
>    The search goes on to balance flexibility, cost, and benefits, as
> always.
>
>    Erik.
>
>> -----Original Message-----
>> From: linux-lvm-bounces@redhat.com
>> [mailto:linux-lvm-bounces@redhat.com] On Behalf Of Ron Watkins
>> Sent: Tuesday, March 01, 2005 11:10 PM
>> To: Erik@echohome.org; LVM general discussion and development
>> Subject: Re: [linux-lvm] Building up a RAID5 LVM home server (long)
>>
>>
>> This is absolutely insane.   It is among the dumbest designs
>> I've ever seen.
>> You are using HDA as a component in MD0 and MD1, and then
>> using HDA AGAIN as
>> part of MD2 directly, while using it indirectly via MD0 and
>> MD1.   You're
>> going to A) bottleneck on HDA, B) you're going to beat that
>> drive to death,
>> and C) if that drive goes, you are HOSED.  Plus, you are just
>> begging for
>> problems with potential bugs in the RAID driver code.  This
>> whole setup is
>> an INCREDIBLY bad idea.  You're trying to 'be clever' to save
>> yourself some
>> money, and all you're doing is buying trouble.
>>
>> The way RAID5 is meant to work is with disks of approximately
>> the same size.
>> RAID5 is not expandable, unless you have a very expensive hardware
>> controller.  There are algorithms that will let you expand
>> the size of a
>> RAID5 volume, but they have not, to my knowledge, been
>> implemented in open
>> source.   You CANNOT do what you want to do, cheaply.  You
>> can spend a great
>> deal of money to satisfy most of your design parameters, but
>> NOT cheaply.
>> If you want it cheap, use fixed drives of about the same
>> size, and don't
>> think about expansion.  When you're ready to expand, hook up
>> another, bigger
>> RAID and copy your data.   In NO case can you use that
>> hodgepodge of junk
>> drives you've collected.
>>
>> Most of your drives are obsolete.  Keep the biggest one, buy
>> at least two
>> more of the same size, and set up a RAID5 using that.  All
>> this monkeying
>> around to try to extract some last value from drives totally
>> ill-suited for
>> the purpose is going to cost you far, far more than new
>> drives ever could.
>>
>> Hell, keep the smaller ones around, put them into a concatenated LVM2
>> volume, and use them as a backup.  It's not the best backup
>> in the world,
>> but it's better than nothing.
>>
>> Do it right.  This is your data you're trying to save.  You
>> can get very
>> nice 250-gig PATA Western Digital drives for $165 from
>> www.newegg.com.  They
>> are specifically designed for RAID.  Buy 4 and save yourself
>> this massive
>> headache.  If you don't need that much space, buy smaller drives.
>>
>> Or, you can persist in trying to be clever, but it's a
>> virtual *certainty*
>> you're going to lose data if you go this route.  Pay now, or
>> pay later.
>> What's your data worth?
>>
>> <<RON>>
>>
>> ----- Original Message ----- 
>> From: "Erik Ohrnberger" <erik@echohome.org>
>> To: "'LVM general discussion and development'" <linux-lvm@redhat.com>
>> Sent: Tuesday, March 01, 2005 10:27 PM
>> Subject: RE: [linux-lvm] Building up a RAID5 LVM home server (long)
>>
>>
>> >
>> > ... SNIP ...
>> >> >What if the I broke everything into 10 GB pieces, and
>> >> created multiple
>> >> >raid5 sets?  Then I could LVM2 them together and have a large
>> >> >filesystem that way.
>> >> >
>> >> >a=20GB, b=30GB, c=40GB
>> >> >
>> >> >a-1 + b-1 + c-1 = md0 (approx 30 GB storage)
>> >> >a-2 + b-2 + c-2 = md1 (approx 30 GB storage)
>> >> >      b-3 + c-3 = md2 (waiting for one more drive)
>> >> >            c-4 = md3 (waiting for two more drives)
>> >> >
>> >> >
>> >> This is sorta what I do.  But in my opinion the gain of
>> having RAID5
>> >> (over RAID1) is when you get over 3 disks...  at 3 disks you are
>> >> burning 33% for redudnacy... 25% or 20% or  17% sounds
>> better to me.
>> >> I guess if
>> >> you go too far it costs in calculating the parity.
>> >
>> > Overhead: Yea, OK. Nothing is without a price.
>> > I fooled around with various ideas, and came up with this for my
>> > particulars:
>> > (Note, rounded to nearest GB)
>> >
>> > 80 GB /dev/hda 60 GB /dev/hdb 40 GB /dev/hdc 45 GB
>> > /dev/hdd
>> >
>> > GB /dev/md0 (RAID0)
>> > 40 /dev/hdc
>> > 15 /dev/hda1
>> > 55
>> >
>> > /dev/md1 (RAID0)
>> > 45 /dev/hdd
>> > 10 /dev/hda2
>> > 55
>> >
>> > /dev/md2 (RAID5)
>> > 55 /dev/md0
>> > 55 /dev/md1
>> > 55 /dev/hda3
>> > 55 /dev/hdb
>> > 220
>> >
>> > Yea, OK, so like the 220 is a bit optimistic, but should get pretty
>> > close
>> > to
>> > that.
>> >
>> > What do you think?
>> >
>> >
>> > _______________________________________________
>> > linux-lvm mailing list
>> > linux-lvm@redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-> lvm
>> > read the
>> LVM HOW-TO at
>> http://tldp.org/HOWTO/LVM-HOWTO/
>> >
>>
>> _______________________________________________
>> linux-lvm mailing list
>> linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm
>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>>
>
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>