[linux-lvm] lvm and fibre channel

[linux-lvm] lvm and fibre channel

[Florin Andrei]

	We have a big Web project, involving several disk arrays (over 3 TB total
storage space) and around 24 Linux servers.
	On the disk arrays, we need to create aprox. 2600 partitions, which may be
mounted by the Linux servers (but no server can mount more than 16 partitions
at any given moment).
	So here is the problem: the Fibre Channel devices are seen as SCSI devices,
but Linux doesn't support more than 2048 partitions on SCSI disks (max 128
disks, max 16 partitions each).

	Does LVM works over Fibre Channel?
	Does it solves our problem?

-- 
Florin Andrei
mailto:florin@linuxstart.com	http://members.linuxstart.com/~florin/
tel: +40-93-261162

Re: [linux-lvm] lvm and fibre channel

[Andreas Dilger]

Florin writes:
> We have a big Web project, involving several disk arrays (over 3 TB total
> storage space) and around 24 Linux servers.
> 
> On the disk arrays, we need to create aprox. 2600 partitions, which may be
> mounted by the Linux servers (but no server can mount more than 16 partitions
> at any given moment).
> 
> So here is the problem: the Fibre Channel devices are seen as SCSI devices,
> but Linux doesn't support more than 2048 partitions on SCSI disks (max 128
> disks, max 16 partitions each).
> 
> Does LVM works over Fibre Channel?

Someone on the list was trying to get LVM to work on FC, but they were
having problems.  I don't know if they got it working at the end.

LVM will not help you here either, because it currently limits you to
255 LVs, regardless of how many VGs you have, because there is only 1
block major for LVM.

The solution to your problem is to allocate more block major numbers
to SCSI.  It would probably be easier to fix SCSI because it already
handles multiple major numbers, unless there is some built-in limit of
128 disks or 128 * 15 = 1920 (not 2048) partitions.

If you add new major numbers for LVM, then you would need to look at
all of the code to handle multiple major numbers.  This would probably
be a lot of work.  You would also have LVM consistency problems between
the nodes in your system (I don't know if LVM works very well in a
clustered environment at all).

You may want to consider something like GFS (www.globalfilesystem.org I
think), which is designed for exactly this sort of environment (although
I don't know if it is ready for production systems yet).

Cheers, Andreas
-- 
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Re: [linux-lvm] lvm and fibre channel

[Andreas Dilger]

I wrote:
> Florin writes:
> > Does LVM works over Fibre Channel?
> 
> Someone on the list was trying to get LVM to work on FC, but they were
> having problems.  I don't know if they got it working at the end.

My mistake, the person having problems was using I2O, and not FC.  Since
the FC disks show up as SCSI disks, they should be fine.  However, it
may be that LVM doesn't work well with the same disks connected to multiple
systems.

Cheers, Andreas
-- 
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Re: [linux-lvm] lvm and fibre channel

[Michael Lausch]

Andreas Dilger writes:
 > Florin writes:
 > > We have a big Web project, involving several disk arrays (over 3 TB total
 > > storage space) and around 24 Linux servers.
 > > 
 > > On the disk arrays, we need to create aprox. 2600 partitions, which may be
 > > mounted by the Linux servers (but no server can mount more than 16 partitions
 > > at any given moment).
 > > 
 > > So here is the problem: the Fibre Channel devices are seen as SCSI devices,
 > > but Linux doesn't support more than 2048 partitions on SCSI disks (max 128
 > > disks, max 16 partitions each).
 > > 
 > > Does LVM works over Fibre Channel?
 > 
 > Someone on the list was trying to get LVM to work on FC, but they were
 > having problems.  I don't know if they got it working at the end.
 > 
 > LVM will not help you here either, because it currently limits you to
 > 255 LVs, regardless of how many VGs you have, because there is only 1
 > block major for LVM.

is using devfs an option? We are planing to attach a PC to a storage
array (HP XP256) using LVM and devfs over fibre channel next week and
doing some tests (performance, ease of use, etc....)

Re: [linux-lvm] lvm and fibre channel

[Andreas Dilger]

Michael Lausch writes:
> Andreas Dilger writes:
>  > LVM will not help you here either, because it currently limits you to
>  > 255 LVs, regardless of how many VGs you have, because there is only 1
>  > block major for LVM.
> 
> is using devfs an option? We are planing to attach a PC to a storage
> array (HP XP256) using LVM and devfs over fibre channel next week and
> doing some tests (performance, ease of use, etc....)

The problem is that the LVM code is written to only look at 1 major number.
The SCSI code has been written to check multiple major numbers.  Maybe it
is worthwhile to change the LVM code to check a whole block of major
numbers, or alternately get dynamic majors via devfs.  This is not there
yet, so you are stuck with a maximum of 255 LVs and 99 VGs.

Cheers, Andreas
-- 
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert