Re: [linux-lvm] How to best use LVM?

Re: [linux-lvm] How to best use LVM?

[pll+lvm]

I guess I should re-title this "How best to lay out my disk space", 
but that seems rather long for a Subject line :)

As I mentioned in a previous post, I'm playing with fire.  I want to 
combine the concepts of LVM with NBD to get an almost infinitely 
scalable amount of storage.  At some point I expect there's going to 
be a limiting factor which prevents the infinite scalability from 
really being infinite.  I haven't determined whether that limiting 
factor is going to be Linux sw RAID, LVM, NBD, the Linux kernel, or 
the network, but one of them will invariably become a bottle neck :)

My idea can be broken down into basically 2 different types of system:
the Storage Node (SN), and the Access Node (AN).

The SN systems will essentially be nothing more than a system with a 
minimal OS installation on it containing almost nothing more than is 
absolutely necessary.  These systems are nothing more than an OS with 
access to gobs of local disk space.

The AN system(s) will be configured as any system which needs access 
to a lot of disk space.  In general, there will be a 1:many ratio 
between ANs and SNs, but I guess you could increase the number of ANs 
if you wanted to create an HA cluster environment.

The SNs will be configured as NBD servers, exporting their local, 
unused disk space out to the AN.

The AN will have LVM installed and be an NBD client.  It will have 
access to all the exported drive space of all the SNs.

Now, the question is, how to deal with all this disk space.  For the 
particular systems I'm using, the basic disk configuration is 4 80GB 
IDE drives partitioned thusly:

	/dev/hda5	487M	/
	/dev/hda1	130M	/boot
	/dev/hda6	4.9G	/usr
	/dev/hda7	4.9G	/var
	/dev/hda8	66GB	empty

	/dev/hdb	80GB	empty
	/dev/hdc	80GB	empty
	/dev/hdd	80GB	empty

Since each SN will have 3 empty IDE drives and one huge empty 
partition on /dev/hda8, I could potentially export about 306GB
((3 * 80GB) + 66GB)  from each system.  To do that, it would seem 
easiest to export each of the 3 drives and the empty partition as 
separate network block devices for a total of 4 exported devices per 
system.

The other option is to ignore the /dev/hda8 partition.  I could 
reserve that for future use on the local system for whatever.
This would leave 3 exportable devices from each SN.  

I was thinking of using RAID5 across these 3 partitions and exporting 
/dev/md0 out via NBD, which would mean only 1 NBD per SN, which leads 
to a little more managability if the number of SNs were to grow very 
high.  This has 2 advantages:

	- RAID5 at the back end for data integrity
	- minimizes the number of network connections back to the SN
	  from the AN.

	  (for some reason I fear a lot of network activity between 
	   the AN and SNs.  Limiting the number of network block 
	   devices seems like a good idea at this point.)

Since the SNs will only be exporting a single NBD in this 
configurations, I figured that I could then use RAID1 (mirroring) on 
the AN to ensure that if any one node were taken out for some reason, 
I'd still have the ability to access the data contained on it.

Since I plan on using LVM on the AN side, each SN-NBD would become a  
PV.  For every 2 SN-PVs added to the environment, I'd be 
able to create a RAID1 meta-device of those PVs, then add them to a 
VG, and then create LVs out of the new space.

There is obviously a *lot* that could go wrong with this entire house 
of cards.  Which is why I'm sending this post.  I'm hoping some 
others might have some good ideas I haven't thought of yet.

Anyway, any feedback anyone has is more than welcome.  I'd love to 
know what people think about this.  Btw, I know it's probably crazy 
to attempt this, but I have the time and the hardware to do so and 
I'm bored :)

Thanks,
-- 

Seeya,
Paul
--
	It may look like I'm just sitting here doing nothing,
   but I'm really actively waiting for all my problems to go away.

	 If you're not having fun, you're not doing it right!

Re: [linux-lvm] How to best use LVM?

[bscott]

On Fri, 16 Aug 2002, at 11:05am, pll+lvm@lanminds.com wrote:
> Now, the question is, how to deal with all this disk space.  For the 
> particular systems I'm using, the basic disk configuration is 4 80GB 
> IDE drives partitioned thusly:
> 
> 	/dev/hda5	487M	/
> 	/dev/hda1	130M	/boot
> 	/dev/hda6	4.9G	/usr
> 	/dev/hda7	4.9G	/var
> 	/dev/hda8	66GB	empty

  Since you describe the Storage Nodes as being little more than a Network
Block Device, that layout seems wrong to me.  I would aim for a totally
stripped down, absolute bare minimum for the SNs.  Say you fit the whole
system into 100 MB.  So create a 100 MB partition on each local drive.  The
system goes into the 100 MB partition on the first disk.  The other three
are ignored.

  The remaining space on each drive is partitioned, and each partition is
exported separately via NBD.  Do all the storage management (concatenation,
stripping, mirroring, etc.) on the Access Node(s).  That gives you maximum
flexibility -- you could mirror individual drives across SNs, for example.

  For reliability, instead of a single partition for the system on SNs,
mirror it using Linux software RAID.

  To make things really interesting, create a four-way mirror of the system
partition on each SN.  That way, an SN would not be completely dead unless
all four disks were dead.  Modifying the Linux software RAID code to support
a four-way mirror is left as an exercise to the reader.

  Actually, if the SN system is simple enough, you could prolly just mirror
it "by hand" -- write the configuration state to each of the four drives.  
Software updates would be a pain, though.  Might just be easier to enable 
4-way mirror (see above).

- 
Ben Scott <bscott@ntisys.com>
| The opinions expressed in this message are those of the author and do not |
| necessarily represent the views or policy of any other person, entity or  |
| organization.  All information is provided without warranty of any kind.  |

[linux-lvm] LVM on top of a 3ware 7500-12 RAID5 set.

[Erik Bussink]

Well Paul,

You seems to want to add a lot of building blocks... I myself am planning
a structure with a single tower case (could be racked in 4U) with a
3ware Escalade 7500-12 which is capable of making a 12 Disk RAID5
volume. I'll probably do a 10x drives, 1x parity and 1x hotspare using
160Gb or 200Gb disk. (The OS is on a mirrored SCSI disk).

But when I start I'll probably only get 5 disk, and add disks to the system
as my need grow. This is where the LVM will help me expand the
LVs.

I decided not to use ReiserFS, but I'm still pondering between Ext3 or
SGI's XFS. As the data on the Logical Volume will be large media
files.

On a side note, the 3ware Escalade supports spin-up controls for
disk supporting it (only IBM drives have it). So you can slowly start
a system without blowing the power supply.

Erik

Re: [linux-lvm] LVM on top of a 3ware 7500-12 RAID5 set.

[Jason L Tibbitts III]

>>>>> "EB" == Erik Bussink <mailinglist@bussink.ch> writes:

EB> But when I start I'll probably only get 5 disk, and add disks to
EB> the system as my need grow. This is where the LVM will help me
EB> expand the LVs.

You can't just add a disk to a RAID5 set, though, unless 3ware has
added that feature very recently.  You can add a whole new RAID set
and add it to your volume group, of course.

 - J<

Re: [linux-lvm] LVM on top of a 3ware 7500-12 RAID5 set.

[Lewis Bergman]

Erik Bussink said:
> Well Paul,
>
> You seems to want to add a lot of building blocks... I myself am
> planning a structure with a single tower case (could be racked in 4U)
> with a 3ware Escalade 7500-12 which is capable of making a 12 Disk RAID5
> volume. I'll probably do a 10x drives, 1x parity and 1x hotspare using
> 160Gb or 200Gb disk. (The OS is on a mirrored SCSI disk).
Could you let us know what kind of disk carriers you use? I have been
looking to do this in a rackmount for a while but haven't been able to
find a decently priced hot-swappable disk carrier yet.
-- 
--
Lewis Bergman
Texas Communications
4309 Maple ST.
Abilene, TX 79602
915-695-6962

Re: [linux-lvm] LVM on top of a 3ware 7500-12 RAID5 set.

[Jason L Tibbitts III]

>>>>> "LB" == Lewis Bergman <lbergman@abi.tconline.net> writes:

LB> Could you let us know what kind of disk carriers you use? I have
LB> been looking to do this in a rackmount for a while but haven't
LB> been able to find a decently priced hot-swappable disk carrier
LB> yet.

Of course 3ware sells 2-space, 3-disk carriers, but they cost about
$200 apiece.  I have several and they are extremely sturdy.
Everything is metal, ventilation is very good and quality is quite
high.  Unfortunately the price can be a bit much.  I'd still recommend
them.

If you're looking to rack things, I recommend just buying a case that
already has the bays installed.  I've used rackmountnet.com
(specifically the 16-disk 4U cases) and have been satisfied overall.
Some components of the bays are plastic and the design requires
adjustment of the bays and receiver for the specific drive type.
Since they stuff five drives in three (half-height) spaces, things are
much more cramped.  I did have a problem with one carrier not sliding
smoothly in one of the cases.

BTW, we discuss things like this at linux-ide-arrays@lists.math.uh.edu
(ping linux-ide-arrays-subscribe@lists.math.uh.edu to join).  It may
be a more appropriate forum than this list.

 - J<

Re: [linux-lvm] LVM on top of a 3ware 7500-12 RAID5 set.

[Jon Bendtsen]

Jason L Tibbitts III wrote:
> 
> >>>>> "EB" == Erik Bussink <mailinglist@bussink.ch> writes:
> 
> EB> But when I start I'll probably only get 5 disk, and add disks to
> EB> the system as my need grow. This is where the LVM will help me
> EB> expand the LVs.
> 
> You can't just add a disk to a RAID5 set, though, unless 3ware has
> added that feature very recently.  You can add a whole new RAID set
> and add it to your volume group, of course.

He could run software raid ontop of it, maybe that will give him more
options as to change the size of the array



JonB

Re: [linux-lvm] LVM on top of a 3ware 7500-12 RAID5 set.

[Jon Bendtsen]

Jason L Tibbitts III wrote:
> 
> >>>>> "LB" == Lewis Bergman <lbergman@abi.tconline.net> writes:
> 
> LB> Could you let us know what kind of disk carriers you use? I have
> LB> been looking to do this in a rackmount for a while but haven't
> LB> been able to find a decently priced hot-swappable disk carrier
> LB> yet.
> 
> Of course 3ware sells 2-space, 3-disk carriers, but they cost about
> $200 apiece.  I have several and they are extremely sturdy.
> Everything is metal, ventilation is very good and quality is quite
> high.  Unfortunately the price can be a bit much.  I'd still recommend
> them.

Ventilation is good. Remember that a HD's MTBF falls to half if the
temperature raises 3 degrees.

 

JonB

Re: [linux-lvm] How to best use LVM?

[pll+lvm]

In a message dated: Fri, 16 Aug 2002 17:48:17 EDT
bscott@ntisys.com said:

>On Fri, 16 Aug 2002, at 11:05am, pll+lvm@lanminds.com wrote:
>> Now, the question is, how to deal with all this disk space.  For the 
>> particular systems I'm using, the basic disk configuration is 4 80GB 
>> IDE drives partitioned thusly:
>> 
>> 	/dev/hda5	487M	/
>> 	/dev/hda1	130M	/boot
>> 	/dev/hda6	4.9G	/usr
>> 	/dev/hda7	4.9G	/var
>> 	/dev/hda8	66GB	empty
>
>  Since you describe the Storage Nodes as being little more than a Network
>Block Device, that layout seems wrong to me.  I would aim for a totally
>stripped down, absolute bare minimum for the SNs.  Say you fit the whole
>system into 100 MB.  So create a 100 MB partition on each local drive.  The
>system goes into the 100 MB partition on the first disk.  The other three
>are ignored.

Well, yes and no.  See, these other SNs, are also pulling double duty 
as servers of various kinds; DNS, web, install server (FAI), etc.  
But those services obviously do not need nearly as much disk space on 
them as they happen to have.  Which is what led me to the idea of 
just using one system as an Access Node to export NFS/CIFS, and 
manage the remainder of the disk space via LVM.

I was planning on mirroring across SNs using Linux's SW RAID, and I 
had considered doing a 4-way mirror of the boot partition on each 
node as well, but I wanted to conquor the NBD-SN-AN idea first :)

-- 

Seeya,
Paul
--
	It may look like I'm just sitting here doing nothing,
   but I'm really actively waiting for all my problems to go away.

	 If you're not having fun, you're not doing it right!