[linux-lvm] LVM limit on PE's/size?

[linux-lvm] LVM limit on PE's/size?

[Scott P]

[-- Attachment #1: Type: text/plain, Size: 1441 bytes --]

Hi all...

I hope I'm not asking a too pettty of a question for this list!

We are planning to add a 3 rd EXP300 array to out current server w/ 2
EXP300's in place already!
We WILL surpass the 1 TB size, if possible?

A little history:

Linux ptb3 2.2.19 #2 SMP Tue Jul 24 11:53:29 CDT 2001 i686 unknown

I have two IBM EXP300 RAID arrays connected to a Netfinity 4500R running
Redhat Linux 6.2.
LVM compiled into kernel AFAIK...w/ patch?

We are using:
lvm_0.9.1_beta7
reiserfsprogs-3.x.0j

Here is the current filesystem and size:

/dev/exp300_2/exp300_lv2 924160464 891307744  32852720      97% /files1

Just wondering about this faq:

                           Q:
                               Why are my logical volumes limited to 256
GB in size?
                           A:
                               This is NO absolute limit but it depends
on the physical extent size you configured at volume group
                               creation time.

                               Please use option -s of the vgcreate
command to give a larger physical extent size. For example with a
                               physical extent size of 524288 KB (512
MB) you are able to map a logical volume of 32 Terabyte.
                               Remember that current Linux kernels are
limited to 1 Terabyte.

***Especially about the part in bold!

Please let me know if you need more details,  I will supply asap!

TIA!

-Scott





[-- Attachment #2: Type: text/html, Size: 2880 bytes --]

Re: [linux-lvm] LVM limit on PE's/size?

[Andreas Dilger]

On Nov 27, 2001  11:34 -0600, Scott P wrote:
> We are planning to add a 3 rd EXP300 array to out current server w/ 2
> EXP300's in place already!  We WILL surpass the 1 TB size, if possible?

Depends if the driver does 32-bit block numbers with signed or unsigned
values.  The absolute maximum supported by 2.4 kernels is 2TB devices.
I don't know if 2.2 will support > 1TB devices, but it is possible.

> Linux ptb3 2.2.19 #2 SMP Tue Jul 24 11:53:29 CDT 2001 i686 unknown
> 
> We are using:
> lvm_0.9.1_beta7
> reiserfsprogs-3.x.0j

Get newer LVM tools, like 1.0.1, because the old ones have lots more bugs.
Beta 7 has the PE alignment bug, so you should upgrade the tools even if
you can't upgrade the kernel.

> Here is the current filesystem and size:
> 
> /dev/exp300_2/exp300_lv2 924160464 891307744  32852720      97% /files1

> Q: Why are my logical volumes limited to 256 GB in size?
> A: This is NO absolute limit but it depends on the physical extent size
> you configured at volume group creation time.

Check what "pvdata -P /dev/whatever" tells you about the PE size.  If
it is 16386kB, then you are limited to 1TB in size.  If it is larger
than that, you _may_ be able to go past 1TB, but you may not, depending
on whether your drivers and kernel work properly or not.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: [linux-lvm] LVM limit on PE's/size?

[Scott P]

Andreas,

Thanks so much for the insight!

Here is some ouput, it looks like we might be ok?
Please comment, if possible, thanks...

-Scott

[root@ptb3 pichelma]# /sbin/pvdisplay /dev/sdb1
--- Physical volume ---
PV Name               /dev/sdb1
VG Name               exp300_2
PV Size               440.71 GB / NOT usable 26.75 MB [LVM: 176 KB]
PV#                   2
PV Status             available
Allocatable           yes (but full)
Cur LV                1
PE Size (KByte)       32768
Total PE              14102
Free PE               0
Allocated PE          14102
PV UUID               1gKwp2-dotn-0WJU-5FwZ-uScI-OyMH-SHVw5F


[root@ptb3 pichelma]# /sbin/pvdisplay /dev/sdc1
--- Physical volume ---
PV Name               /dev/sdc1
VG Name               exp300_2
PV Size               440.71 GB / NOT usable 26.75 MB [LVM: 176 KB]
PV#                   1
PV Status             available
Allocatable           yes (but full)
Cur LV                1
PE Size (KByte)       32768
Total PE              14102
Free PE               0
Allocated PE          14102
PV UUID               ciUWIf-i9OO-okne-Fs6R-fjU1-NPpr-Ml4p8L


[root@ptb3 pichelma]# /sbin/pvdata -P /dev/sdb1
--- Physical volume ---
PV Name               /dev/sdb1
VG Name               exp300_2
PV Size               440.71 GB / NOT usable 26.75 MB [LVM: 176 KB]
PV#                   2
PV Status             available
Allocatable           yes (but full)
Cur LV                1
PE Size (KByte)       32768
Total PE              14102
Free PE               0
Allocated PE          14102
PV UUID               1gKwp2-dotn-0WJU-5FwZ-uScI-OyMH-SHVw5F


[root@ptb3 pichelma]# /sbin/pvdata -P /dev/sdc1
--- Physical volume ---
PV Name               /dev/sdc1
VG Name               exp300_2
PV Size               440.71 GB / NOT usable 26.75 MB [LVM: 176 KB]
PV#                   1
PV Status             available
Allocatable           yes (but full)
Cur LV                1
PE Size (KByte)       32768
Total PE              14102
Free PE               0
Allocated PE          14102
PV UUID               ciUWIf-i9OO-okne-Fs6R-fjU1-NPpr-Ml4p8L



Andreas Dilger wrote:

> On Nov 27, 2001  11:34 -0600, Scott P wrote:
> > We are planning to add a 3 rd EXP300 array to out current server w/ 2
> > EXP300's in place already!  We WILL surpass the 1 TB size, if possible?
>
> Depends if the driver does 32-bit block numbers with signed or unsigned
> values.  The absolute maximum supported by 2.4 kernels is 2TB devices.
> I don't know if 2.2 will support > 1TB devices, but it is possible.
>
> > Linux ptb3 2.2.19 #2 SMP Tue Jul 24 11:53:29 CDT 2001 i686 unknown
> >
> > We are using:
> > lvm_0.9.1_beta7
> > reiserfsprogs-3.x.0j
>
> Get newer LVM tools, like 1.0.1, because the old ones have lots more bugs.
> Beta 7 has the PE alignment bug, so you should upgrade the tools even if
> you can't upgrade the kernel.
>
> > Here is the current filesystem and size:
> >
> > /dev/exp300_2/exp300_lv2 924160464 891307744  32852720      97% /files1
>
> > Q: Why are my logical volumes limited to 256 GB in size?
> > A: This is NO absolute limit but it depends on the physical extent size
> > you configured at volume group creation time.
>
> Check what "pvdata -P /dev/whatever" tells you about the PE size.  If
> it is 16386kB, then you are limited to 1TB in size.  If it is larger
> than that, you _may_ be able to go past 1TB, but you may not, depending
> on whether your drivers and kernel work properly or not.
>
> Cheers, Andreas
> --
> Andreas Dilger
> http://sourceforge.net/projects/ext2resize/
> http://www-mddsp.enel.ucalgary.ca/People/adilger/
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html

Re: [linux-lvm] LVM limit on PE's/size?

[Heinz J . Mauelshagen]

Scott,

with your PE size of 32MB, you can achieve to create LVs up to 2TB in size
(actually it is a little bit less than 2TB, because every LV can have up
to 64K - 2 extents). You can have multiple 2TB LVs this way, because your
VG capacity is in no means limited to 2TB! In theory you could have 128 SCSI
IDs with 8 LUNs each providing up to 1 or 2TB in 1 VG.

Because your PVs are smaller than 1TB in size, you will not suffer from
the block adrdessing sign bit danger Andreas was explaining in his email.

In general you should plan for the largest possible Logical Volume size
during the lifetime of a Volume Group and set the Physical Extent size
(vgcreate -s) acordingly. Even though Linux does not support more than 2TB
per block device today, it will be in the future and you won't suffer from
the LVM1 constraint ITR.

Regards,
Heinz    -- The LVM Guy --


On Tue, Nov 27, 2001 at 02:00:07PM -0600, Scott P wrote:
> Andreas,
> 
> Thanks so much for the insight!
> 
> Here is some ouput, it looks like we might be ok?
> Please comment, if possible, thanks...
> 
> -Scott
> 
> [root@ptb3 pichelma]# /sbin/pvdisplay /dev/sdb1
> --- Physical volume ---
> PV Name               /dev/sdb1
> VG Name               exp300_2
> PV Size               440.71 GB / NOT usable 26.75 MB [LVM: 176 KB]
> PV#                   2
> PV Status             available
> Allocatable           yes (but full)
> Cur LV                1
> PE Size (KByte)       32768
> Total PE              14102
> Free PE               0
> Allocated PE          14102
> PV UUID               1gKwp2-dotn-0WJU-5FwZ-uScI-OyMH-SHVw5F
> 
> 
> [root@ptb3 pichelma]# /sbin/pvdisplay /dev/sdc1
> --- Physical volume ---
> PV Name               /dev/sdc1
> VG Name               exp300_2
> PV Size               440.71 GB / NOT usable 26.75 MB [LVM: 176 KB]
> PV#                   1
> PV Status             available
> Allocatable           yes (but full)
> Cur LV                1
> PE Size (KByte)       32768
> Total PE              14102
> Free PE               0
> Allocated PE          14102
> PV UUID               ciUWIf-i9OO-okne-Fs6R-fjU1-NPpr-Ml4p8L
> 
> 
> [root@ptb3 pichelma]# /sbin/pvdata -P /dev/sdb1
> --- Physical volume ---
> PV Name               /dev/sdb1
> VG Name               exp300_2
> PV Size               440.71 GB / NOT usable 26.75 MB [LVM: 176 KB]
> PV#                   2
> PV Status             available
> Allocatable           yes (but full)
> Cur LV                1
> PE Size (KByte)       32768
> Total PE              14102
> Free PE               0
> Allocated PE          14102
> PV UUID               1gKwp2-dotn-0WJU-5FwZ-uScI-OyMH-SHVw5F
> 
> 
> [root@ptb3 pichelma]# /sbin/pvdata -P /dev/sdc1
> --- Physical volume ---
> PV Name               /dev/sdc1
> VG Name               exp300_2
> PV Size               440.71 GB / NOT usable 26.75 MB [LVM: 176 KB]
> PV#                   1
> PV Status             available
> Allocatable           yes (but full)
> Cur LV                1
> PE Size (KByte)       32768
> Total PE              14102
> Free PE               0
> Allocated PE          14102
> PV UUID               ciUWIf-i9OO-okne-Fs6R-fjU1-NPpr-Ml4p8L
> 
> 
> 
> Andreas Dilger wrote:
> 
> > On Nov 27, 2001  11:34 -0600, Scott P wrote:
> > > We are planning to add a 3 rd EXP300 array to out current server w/ 2
> > > EXP300's in place already!  We WILL surpass the 1 TB size, if possible?
> >
> > Depends if the driver does 32-bit block numbers with signed or unsigned
> > values.  The absolute maximum supported by 2.4 kernels is 2TB devices.
> > I don't know if 2.2 will support > 1TB devices, but it is possible.
> >
> > > Linux ptb3 2.2.19 #2 SMP Tue Jul 24 11:53:29 CDT 2001 i686 unknown
> > >
> > > We are using:
> > > lvm_0.9.1_beta7
> > > reiserfsprogs-3.x.0j
> >
> > Get newer LVM tools, like 1.0.1, because the old ones have lots more bugs.
> > Beta 7 has the PE alignment bug, so you should upgrade the tools even if
> > you can't upgrade the kernel.
> >
> > > Here is the current filesystem and size:
> > >
> > > /dev/exp300_2/exp300_lv2 924160464 891307744  32852720      97% /files1
> >
> > > Q: Why are my logical volumes limited to 256 GB in size?
> > > A: This is NO absolute limit but it depends on the physical extent size
> > > you configured at volume group creation time.
> >
> > Check what "pvdata -P /dev/whatever" tells you about the PE size.  If
> > it is 16386kB, then you are limited to 1TB in size.  If it is larger
> > than that, you _may_ be able to go past 1TB, but you may not, depending
> > on whether your drivers and kernel work properly or not.
> >
> > Cheers, Andreas
> > --
> > Andreas Dilger
> > http://sourceforge.net/projects/ext2resize/
> > http://www-mddsp.enel.ucalgary.ca/People/adilger/
> >
> > _______________________________________________
> > linux-lvm mailing list
> > linux-lvm@sistina.com
> > http://lists.sistina.com/mailman/listinfo/linux-lvm
> > read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
> 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@Sistina.com                           +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Re: [linux-lvm] LVM limit on PE's/size?

[Scott P]

Heinz,

Thanks for clarifying further, I appreciate the extra info!

-Scott


"Heinz J . Mauelshagen" wrote:

> Scott,
>
> with your PE size of 32MB, you can achieve to create LVs up to 2TB in size
> (actually it is a little bit less than 2TB, because every LV can have up
> to 64K - 2 extents). You can have multiple 2TB LVs this way, because your
> VG capacity is in no means limited to 2TB! In theory you could have 128 SCSI
> IDs with 8 LUNs each providing up to 1 or 2TB in 1 VG.
>
> Because your PVs are smaller than 1TB in size, you will not suffer from
> the block adrdessing sign bit danger Andreas was explaining in his email.
>
> In general you should plan for the largest possible Logical Volume size
> during the lifetime of a Volume Group and set the Physical Extent size
> (vgcreate -s) acordingly. Even though Linux does not support more than 2TB
> per block device today, it will be in the future and you won't suffer from
> the LVM1 constraint ITR.
>
> Regards,
> Heinz    -- The LVM Guy --
>
> On Tue, Nov 27, 2001 at 02:00:07PM -0600, Scott P wrote:
> > Andreas,
> >
> > Thanks so much for the insight!
> >
> > Here is some ouput, it looks like we might be ok?
> > Please comment, if possible, thanks...
> >
> > -Scott
> >
> > [root@ptb3 pichelma]# /sbin/pvdisplay /dev/sdb1
> > --- Physical volume ---
> > PV Name               /dev/sdb1
> > VG Name               exp300_2
> > PV Size               440.71 GB / NOT usable 26.75 MB [LVM: 176 KB]
> > PV#                   2
> > PV Status             available
> > Allocatable           yes (but full)
> > Cur LV                1
> > PE Size (KByte)       32768
> > Total PE              14102
> > Free PE               0
> > Allocated PE          14102
> > PV UUID               1gKwp2-dotn-0WJU-5FwZ-uScI-OyMH-SHVw5F
> >
> >
> > [root@ptb3 pichelma]# /sbin/pvdisplay /dev/sdc1
> > --- Physical volume ---
> > PV Name               /dev/sdc1
> > VG Name               exp300_2
> > PV Size               440.71 GB / NOT usable 26.75 MB [LVM: 176 KB]
> > PV#                   1
> > PV Status             available
> > Allocatable           yes (but full)
> > Cur LV                1
> > PE Size (KByte)       32768
> > Total PE              14102
> > Free PE               0
> > Allocated PE          14102
> > PV UUID               ciUWIf-i9OO-okne-Fs6R-fjU1-NPpr-Ml4p8L
> >
> >
> > [root@ptb3 pichelma]# /sbin/pvdata -P /dev/sdb1
> > --- Physical volume ---
> > PV Name               /dev/sdb1
> > VG Name               exp300_2
> > PV Size               440.71 GB / NOT usable 26.75 MB [LVM: 176 KB]
> > PV#                   2
> > PV Status             available
> > Allocatable           yes (but full)
> > Cur LV                1
> > PE Size (KByte)       32768
> > Total PE              14102
> > Free PE               0
> > Allocated PE          14102
> > PV UUID               1gKwp2-dotn-0WJU-5FwZ-uScI-OyMH-SHVw5F
> >
> >
> > [root@ptb3 pichelma]# /sbin/pvdata -P /dev/sdc1
> > --- Physical volume ---
> > PV Name               /dev/sdc1
> > VG Name               exp300_2
> > PV Size               440.71 GB / NOT usable 26.75 MB [LVM: 176 KB]
> > PV#                   1
> > PV Status             available
> > Allocatable           yes (but full)
> > Cur LV                1
> > PE Size (KByte)       32768
> > Total PE              14102
> > Free PE               0
> > Allocated PE          14102
> > PV UUID               ciUWIf-i9OO-okne-Fs6R-fjU1-NPpr-Ml4p8L
> >
> >
> >
> > Andreas Dilger wrote:
> >
> > > On Nov 27, 2001  11:34 -0600, Scott P wrote:
> > > > We are planning to add a 3 rd EXP300 array to out current server w/ 2
> > > > EXP300's in place already!  We WILL surpass the 1 TB size, if possible?
> > >
> > > Depends if the driver does 32-bit block numbers with signed or unsigned
> > > values.  The absolute maximum supported by 2.4 kernels is 2TB devices.
> > > I don't know if 2.2 will support > 1TB devices, but it is possible.
> > >
> > > > Linux ptb3 2.2.19 #2 SMP Tue Jul 24 11:53:29 CDT 2001 i686 unknown
> > > >
> > > > We are using:
> > > > lvm_0.9.1_beta7
> > > > reiserfsprogs-3.x.0j
> > >
> > > Get newer LVM tools, like 1.0.1, because the old ones have lots more bugs.
> > > Beta 7 has the PE alignment bug, so you should upgrade the tools even if
> > > you can't upgrade the kernel.
> > >
> > > > Here is the current filesystem and size:
> > > >
> > > > /dev/exp300_2/exp300_lv2 924160464 891307744  32852720      97% /files1
> > >
> > > > Q: Why are my logical volumes limited to 256 GB in size?
> > > > A: This is NO absolute limit but it depends on the physical extent size
> > > > you configured at volume group creation time.
> > >
> > > Check what "pvdata -P /dev/whatever" tells you about the PE size.  If
> > > it is 16386kB, then you are limited to 1TB in size.  If it is larger
> > > than that, you _may_ be able to go past 1TB, but you may not, depending
> > > on whether your drivers and kernel work properly or not.
> > >
> > > Cheers, Andreas
> > > --
> > > Andreas Dilger
> > > http://sourceforge.net/projects/ext2resize/
> > > http://www-mddsp.enel.ucalgary.ca/People/adilger/
> > >
> > > _______________________________________________
> > > linux-lvm mailing list
> > > linux-lvm@sistina.com
> > > http://lists.sistina.com/mailman/listinfo/linux-lvm
> > > read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
> >
> >
> > _______________________________________________
> > linux-lvm mailing list
> > linux-lvm@sistina.com
> > http://lists.sistina.com/mailman/listinfo/linux-lvm
> > read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
>
> *** Software bugs are stupid.
>     Nevertheless it needs not so stupid people to solve them ***
>
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>
> Heinz Mauelshagen                                 Sistina Software Inc.
> Senior Consultant/Developer                       Am Sonnenhang 11
>                                                   56242 Marienrachdorf
>                                                   Germany
> Mauelshagen@Sistina.com                           +49 2626 141200
>                                                        FAX 924446
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html

Re: [linux-lvm] LVM limit on PE's/size?

[Peter Skensved]

      
>In general you should plan for the largest possible Logical Volume size
>during the lifetime of a Volume Group and set the Physical Extent size
>(vgcreate -s) acordingly. Even though Linux does not support more than 2TB
>per block device today, it will be in the future and you won't suffer from
>the LVM1 constraint ITR.
>
>Regards,
>Heinz    -- The LVM Guy --
>

    Is there any way to change the PE size `after the fact ' ?  I have a
255.99 GB logical volume ( three 80 GB drives ) with PE size set to 4 Mb.
Is there a non-destructive way to increase the PE size so that I can add
more drives ? Or do I have to back everything up and start from scratch ?
                                                   peter

----

Peter Skensved                          Email : peter@SNO.Phy.QueensU.CA
Dept. of Physics,                       Phone:  (613) 533-2676
Queen's University,                     Fax:    (613) 533-6813
Kingston, Ontario,
Canada

Re: [linux-lvm] LVM limit on PE's/size?

[Andreas Dilger]

On Nov 29, 2001  11:40 -0500, Peter Skensved wrote:
>       
> >In general you should plan for the largest possible Logical Volume size
> >during the lifetime of a Volume Group and set the Physical Extent size
> >(vgcreate -s) acordingly. Even though Linux does not support more than 2TB
> >per block device today, it will be in the future and you won't suffer from
> >the LVM1 constraint ITR.
> 
>     Is there any way to change the PE size `after the fact ' ?
 
 No.

> I have a 255.99 GB logical volume ( three 80 GB drives ) with PE size set
> to 4 Mb. Is there a non-destructive way to increase the PE size so that I
> can add more drives ? Or do I have to back everything up and start from
> scratch ?

Well, if you are adding more drives, you could always create a new VG with
the larger PE size, copy over all of your data, and then add the old drives
into the new VG after removing them from the old VG.  Sadly, the PE size
seems to be stored in the PV header (as well as the VG header), so it may
be that you need to re-initialize the PV to change the PE size (maybe not,
I don't know).

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/