[linux-lvm] Possible to raise Max LV?

[linux-lvm] Possible to raise Max LV?

[Dave Alden]

Hi,
  I'm hoping to use LVM with my hardware RAID array so I can take snapshots
of the filesystem for backups.  What I'm worried about is that my current
array is 240GB, so the next drive I add will take it over 256GB which (if
I'm reading the HOWTO correctly) means that I won't be able to use LVM
anymore (since it's being presented as one drive).  Is there someway to
get around the 256GB limit (or did I misunderstand this completely? :-)
...thnx,
...dave alden

Re: [linux-lvm] Possible to raise Max LV?

[Andreas Dilger]

Dave Alden writes:
>   I'm hoping to use LVM with my hardware RAID array so I can take snapshots
> of the filesystem for backups.  What I'm worried about is that my current
> array is 240GB, so the next drive I add will take it over 256GB which (if
> I'm reading the HOWTO correctly) means that I won't be able to use LVM
> anymore (since it's being presented as one drive).  Is there someway to
> get around the 256GB limit (or did I misunderstand this completely? :-)

The max LV size is dependent upon the PE size that you select at VG
creation time.  This is because there is a limit of 64k PEs per LV.

64k = 2^16, 4MB = 2^22 -> 2^16 * 2^22 = 2^38 = 2^8 * 2^30 = 256GB

There is no way to fix this, unfortunately, without backup/restore or
having enough disk space to make a new VG with larger PE size and
then copy your filesystem over.  You can only have 1 PE size in a VG,
so it is not possible to do a "pvmigrate" to a larger PE size.

If you don't have a single LV > 256GB then you do not have a problem.
The 256GB limit is per LV.

Cheers, Andreas
-- 
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Re: [linux-lvm] Possible to raise Max LV?

[Dave Alden]

Hi,
  First let me apologize -- I just realized my question was in the FAQ.
I'm usually brighter than this, honest.  :-)  Now on to my new question:

On Tue, Feb 27, 2001 at 09:58:46AM -0700, Andreas Dilger wrote:
> The max LV size is dependent upon the PE size that you select at VG
> creation time.  This is because there is a limit of 64k PEs per LV.

I'm completely new to LVM, so can someone explain to me the purpose of
the PE's and why they default to 4M?  Is there some kind of performance
or space hit if I raise that to 16M (remember my setup: hardware RAID
array that is currently at ~250G, soon to be ~325G)?

...thnx,
...dave alden

Re: [linux-lvm] Possible to raise Max LV?

[Jay Weber]

I gather 4M was just a default starting point.  I personally use 64M so I
can theoretically grow to at least 2TB in the future or beyond even. :)

I have several JBODs in the back that are 500G each and ideally I'd like
to merge a few together and hit 1TB and 2TB filesystems when the time is
right.

I've seen no real difference in performance regarding the PE size myself,
granted large PE's means larger chunks of allocation when you're adding
PE's to a LV, etc.

On Tue, 27 Feb 2001, Dave Alden wrote:

> Date: Tue, 27 Feb 2001 13:23:58 -0500
> From: Dave Alden <alden@math.ohio-state.edu>
> Reply-To: linux-lvm@sistina.com
> To: linux-lvm@sistina.com
> Subject: Re: [linux-lvm] Possible to raise Max LV?
>
> Hi,
>   First let me apologize -- I just realized my question was in the FAQ.
> I'm usually brighter than this, honest.  :-)  Now on to my new question:
>
> On Tue, Feb 27, 2001 at 09:58:46AM -0700, Andreas Dilger wrote:
> > The max LV size is dependent upon the PE size that you select at VG
> > creation time.  This is because there is a limit of 64k PEs per LV.
>
> I'm completely new to LVM, so can someone explain to me the purpose of
> the PE's and why they default to 4M?  Is there some kind of performance
> or space hit if I raise that to 16M (remember my setup: hardware RAID
> array that is currently at ~250G, soon to be ~325G)?
>
> ...thnx,
> ...dave alden
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
>

Re: [linux-lvm] Possible to raise Max LV?

[Andreas Dilger]

Dave Alden writes:
> I'm completely new to LVM, so can someone explain to me the purpose of
> the PE's and why they default to 4M?  Is there some kind of performance
> or space hit if I raise that to 16M (remember my setup: hardware RAID
> array that is currently at ~250G, soon to be ~325G)?

Basically, the PE size is the minimum allocation unit of LVM.  If you
make PE size very large, then you are generally wasting PE/2 disk space
for each PV.  If (as in your case) the HW raid appears as a single large
disk, then a large PE size may be appropriate, since you will only waste
PE/2 once and it will still let you grow the LV to the current 2TB block
device limit (kernel limit).

However, LVM DOES NOT handle PVs that change size!!!  In this case you
need to make the new disk space appear as a new PV (via a new LUN number
or something) and add that PV to the VG.  Sorry.

Cheers, Andreas
-- 
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Re: [linux-lvm] Possible to raise Max LV?

[Ragnar Kjørstad]

On Tue, Feb 27, 2001 at 01:22:18PM -0700, Andreas Dilger wrote:
> Basically, the PE size is the minimum allocation unit of LVM.  If you
> make PE size very large, then you are generally wasting PE/2 disk space
> for each PV.  If (as in your case) the HW raid appears as a single large
> disk, then a large PE size may be appropriate, since you will only waste
> PE/2 once and it will still let you grow the LV to the current 2TB block
> device limit (kernel limit).

What about increesing the default, to say 32 MB?
32 MB should be enough for everyone (because of the 2TB limit), and 16
MB waste on every PV isn't that bad.



-- 
Ragnar Kj�rstad
Big Storage

Re: [linux-lvm] Possible to raise Max LV?

[Andreas Dilger]

Ragnar Kj_rstad writes:
> What about increesing the [PE size] default, to say 32 MB?
> 32 MB should be enough for everyone (because of the 2TB limit), and 16
> MB waste on every PV isn't that bad.

So says the guy from a company called "Big Storage" ;-).  However, I've
seen several people who make multiple partitions on a single disk, and
then turn these into PVs (don't ask me why), so they would be wasting
16MB per partition.  Most people will not need 2TB LVs for some time to
come.  However, if your systems usually have large LVs, then it is OK
for you to increase the default for your systems.

Cheers, Andreas
-- 
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Re: [linux-lvm] Possible to raise Max LV?

[Eric M. Hopper]

[-- Attachment #1: Type: text/plain, Size: 1489 bytes --]

On Tue, Feb 27, 2001 at 01:55:05PM -0700, Andreas Dilger wrote:
> Ragnar Kj_rstad writes:
> > What about increesing the [PE size] default, to say 32 MB?
> > 32 MB should be enough for everyone (because of the 2TB limit), and 16
> > MB waste on every PV isn't that bad.
> 
> So says the guy from a company called "Big Storage" ;-).  However,
> I've seen several people who make multiple partitions on a single
> disk, and then turn these into PVs (don't ask me why), so they would
> be wasting 16MB per partition.  Most people will not need 2TB LVs for
> some time to come.  However, if your systems usually have large LVs,
> then it is OK for you to increase the default for your systems.

	I actually intend to for mine sometime later because it reduces
the fragmentation problem.  I know it shouldn't be that big a deal with
4M PEs, but with 32M PEs it'll be even less of a deal.  :-)

	I actually thinking of generally scaling my PEs to around 0.1%
of my total available space on the theory that allocation of finer
grained chunks than that is kinda silly.

	If I weren't working on some stuff of my own, I'd dive in and
see if I could write a defrag utility for LVM.  The part I'm not sure
how I'd handle would be striped LVs.

Have fun (if at all possible),
-- 
The best we can hope for concerning the people at large is that they
be properly armed.  -- Alexander Hamilton
-- Eric Hopper (hopper@omnifarious.mn.org http://www.omnifarious.org/~hopper) --

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: [linux-lvm] Possible to raise Max LV?

[Heinz J. Mauelshagen]

On Tue, Feb 27, 2001 at 03:04:17PM -0600, Eric M. Hopper wrote:
> On Tue, Feb 27, 2001 at 01:55:05PM -0700, Andreas Dilger wrote:
> > Ragnar Kj_rstad writes:
> > > What about increesing the [PE size] default, to say 32 MB?
> > > 32 MB should be enough for everyone (because of the 2TB limit), and 16
> > > MB waste on every PV isn't that bad.
> > 
> > So says the guy from a company called "Big Storage" ;-).  However,
> > I've seen several people who make multiple partitions on a single
> > disk, and then turn these into PVs (don't ask me why), so they would
> > be wasting 16MB per partition.  Most people will not need 2TB LVs for
> > some time to come.  However, if your systems usually have large LVs,
> > then it is OK for you to increase the default for your systems.
> 
> 	I actually intend to for mine sometime later because it reduces
> the fragmentation problem.  I know it shouldn't be that big a deal with
> 4M PEs, but with 32M PEs it'll be even less of a deal.  :-)
> 
> 	I actually thinking of generally scaling my PEs to around 0.1%
> of my total available space on the theory that allocation of finer
> grained chunks than that is kinda silly.
> 
> 	If I weren't working on some stuff of my own, I'd dive in and
> see if I could write a defrag utility for LVM.  The part I'm not sure
> how I'd handle would be striped LVs.
> 
> Have fun (if at all possible),
> -- 

As Andreas already pointed out: 4M basically is a good value to keep
the overhead small.

As I stated in another email: the ~64k extents per LV limit will likely
disappear in the future (no date though ;-).
For now you *need* to plan for the maximum LV size you want to create/extend to
during planning of the VG(s). 2T is the Linux limit per block device anyway
at least until Linux 2.5 changes that.

-- 

Regards,
Heinz    -- The LVM Guy --

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@Sistina.com                           +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Re: [linux-lvm] Possible to raise Max LV?

[zoo1]

[-- Attachment #1: Type: text/plain, Size: 1031 bytes --]

On Tue, Feb 27, 2001 at 01:55:05PM -0700, Andreas Dilger wrote:

>So says the guy from a company called "Big Storage" ;-).  However, I've
>seen several people who make multiple partitions on a single disk, and
>then turn these into PVs (don't ask me why),

easy way to convert an existing single-disk installation to LVM. make
the free space (hope you've got some) into a PV, create LVs, move over
your filesystem mountpoint by mountpoint and make the old partitions
into PVs as you go so you don't run out of disk. i was seriously
considering going that way myself.

then again, i was also considering creating two VGs, one for "user" data
and one for "system" stuff. now, i'm trying to figure out what i thought
the point of this was, maybe something about possible future migrations
to multispindle systems.

-- 
   PGP/GnuPG key (ID 1024D/BFE0D6D0) available from keyservers everywhere

       "Everything I am today, I owe to people whom it is now too late
                                   to punish."

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: [linux-lvm] Possible to raise Max LV?

[Andreas Dilger]

"zoo1" writes:
> i was also considering creating two VGs, one for "user" data
> and one for "system" stuff. now, i'm trying to figure out what i thought
> the point of this was, maybe something about possible future migrations
> to multispindle systems.

One reason to do this (on AIX) at least was that for production systems
it was useful to be able to move a VG to another server if the original
server died a horrible death.  Also, if you have server failover (HA),
you need to have all of the data in a single VG in order to import it
on the backup server (of course a replicated distributed filesystem is
better).

Finally, on AIX, you have a backup tool called "mksysb" which would give
you a backup tape so that you can install a new/replacement system from
the tape, and it configures all of your LVs for you etc.  The mksysb
backup only backs up the "rootvg", which normally holds /, /usr, /tmp,
/var, /home, /boot, and swap (swap and /tmp are not backed up, but are
re-created at install time).  If you put too much stuff in your rootvg
then it was in danger of not fitting on a single tape, and you also may
restore a lot of junk you don't want on another system.

I've been thinking about doing something like mksysb for Linux, but I
need a bit more support in ext2 (to hold the last mounted directory
for rebuilding /etc/fstab).  Other than that, it would be supremely
convenient for bare-metal restores of a system (only need a boot floppy).

Cheers, Andreas
-- 
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Re: [linux-lvm] Possible to raise Max LV?

[Steven Lembark]

> I'm completely new to LVM, so can someone explain to me the purpose of
> the PE's and why they default to 4M?  Is there some kind of performance
> or space hit if I raise that to 16M (remember my setup: hardware RAID
> array that is currently at ~250G, soon to be ~325G)?

also described in the docs.  

you might be better off splitting the space into smaller chunks.
with that much space in one unit you will be hit hard if any of
it fries -- backups or not.  do you have any files that really
take up 325G?  if not you could split the space into multiple
128GB sections for separate use.  makes backups a whole lot 
easier...  let alone restores.

-- 
 Steven Lembark                                   2930 W. Palmer St.
                                                 Chicago, IL  60647
 lembark@wrkhors.com                                   800-762-1582

Re: [linux-lvm] Possible to raise Max LV?

[Heinz J. Mauelshagen]

On Tue, Feb 27, 2001 at 11:07:45PM -0600, Steven Lembark wrote:
> 
> > I'm completely new to LVM, so can someone explain to me the purpose of
> > the PE's and why they default to 4M?  Is there some kind of performance
> > or space hit if I raise that to 16M (remember my setup: hardware RAID
> > array that is currently at ~250G, soon to be ~325G)?
> 

Linux LVM follows the paradigm of block oriented (like the AIX or HP ones)
rather than section oriented Volume Managers (like the Veritas one).

Therefore it deals with so called Physical Extents (or PEs) as the basic
allocation units. The actual LVM code supports up to ~64k extents per
Logical Volume. If you like to have Logical Volumes larger than ~256G in the
future you *need* to create your VG(s) with a PE size larger than the default
of 4M.

Even though that LVM constraint most likely will disappear in the future
you should plan for larger PE sizes now in case 256G per LV maximum size
is too small for you.

If you raise the PE size to 16M your metadata usage will be a little bit smaller
*but* the amount of free space after the metadata will likely be larger as
with 4M PE size due to the allocation policy of LVM. This choice will give
you a maximum LV size of 1T. Linux is limited to 2T per single block device
at this point in time. In Linux 2.5 that constraint will disapear too.

> also described in the docs.  
> 
> you might be better off splitting the space into smaller chunks.
> with that much space in one unit you will be hit hard if any of
> it fries -- backups or not.  do you have any files that really
> take up 325G?  if not you could split the space into multiple
> 128GB sections for separate use.  makes backups a whole lot 
> easier...  let alone restores.
> 
> -- 
>  Steven Lembark                                   2930 W. Palmer St.
>                                                  Chicago, IL  60647
>  lembark@wrkhors.com                                   800-762-1582
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm

-- 

Regards,
Heinz    -- The LVM Guy --

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@Sistina.com                           +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-