* [linux-lvm] large logical volumes
@ 2005-03-31 0:29 Guido Vettoretti
2005-04-29 20:00 ` Myrddin Emrys
0 siblings, 1 reply; 5+ messages in thread
From: Guido Vettoretti @ 2005-03-31 0:29 UTC (permalink / raw)
To: linux-lvm
Hello,
I'm trying to set up a large logical volume (13.1 TB) on a RHEL 4.0
server. I have 8 scsi devices of 1.8Tb each.
When I try to do a 'mkfs' and set up a large disk I get limited to 4Tbytes.
This command creates a smaller filesystem then
[root@cyclone ~]# lvcreate -l 1287525 -i 3 -n lvol0 vol0
this one.
[root@cyclone ~]# lvcreate -l 1000000 -i 3 -n lvol0 vol0
It seems that there is a limit on the number of block groups of 32768
and blocks per group of 32768.
My block size is 4096
so I can only get an LV of 4096*32768*32768 = 4TB.
I tried modifying the PE size on the volume group, but I'm still having
problems.
Can anyone help please?
Thanks
Guido
---------------
[root@cyclone ~]# mkfs -t ext3 /dev/vol0/lvol0
mke2fs 1.35 (28-Feb-2004)
max_blocks 4294967295, rsv_groups = 0, rsv_gdb = 1024
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
512016384 inodes, 1024002048 blocks
8250429 blocks (0.81%) reserved for the super user
First data block=0
Maximum filesystem blocks=1027604480
31251 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632,
2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000, 214990848, 512000000, 550731776, 644972544
Here is the entire process I typed:
------------------------------------------------------------------------------
[root@cyclone ~]# pvcreate /dev/sdc1 /dev/sdd1 /dev/sde1
Physical volume "/dev/sdc1" successfully created
Physical volume "/dev/sdd1" successfully created
Physical volume "/dev/sde1" successfully created
[root@cyclone ~]# pvscan
/dev/cdrom: open failed: No medium found
PV /dev/sda2 VG VolGroup00 lvm2 [34.06 GB / 0 free]
PV /dev/sdb1 VG VolGroup00 lvm2 [34.16 GB / 0 free]
PV /dev/sdc1 lvm2 [1.64 TB]
PV /dev/sdd1 lvm2 [1.64 TB]
PV /dev/sde1 lvm2 [1.64 TB]
Total: 5 [4.98 TB] / in use: 2 [68.22 GB] / in no VG: 3 [4.91 TB]
[root@cyclone ~]# vgcreate vol0 /dev/sd[c-e]1
Volume group "vol0" successfully created
[root@cyclone ~]# vgdisplay
--- Volume group ---
VG Name vol0
System ID
Format lvm2
Metadata Areas 3
Metadata Sequence No 1
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 0
Open LV 0
Max PV 0
Cur PV 3
Act PV 3
VG Size 4.91 TB
PE Size 4.00 MB
Total PE 1287525
Alloc PE / Size 0 / 0
Free PE / Size 1287525 / 4.91 TB
VG UUID fQ7cKh-FVnl-NglQ-jimv-RsZK-dU5j-zYKIcG
[root@cyclone ~]# lvcreate -l 1287525 -i 3 -n lvol0 vol0
Using default stripesize 64KB
Logical volume "lvol0" created
[root@cyclone ~]# lvdisplay
--- Logical volume ---
LV Name /dev/vol0/lvol0
VG Name vol0
LV UUID c2MADb-BoAO-lhl3-3ukj-XfyQ-K8L0-cxcIlw
LV Write Access read/write
LV Status available
# open 0
LV Size 4.91 TB
Current LE 1287525
Segments 1
Allocation inherit
Read ahead sectors 0
Block device 253:2
[root@cyclone ~]# mkfs -t ext3 /dev/vol0/lvol0
mke2fs 1.35 (28-Feb-2004)
max_blocks 4294967295, rsv_groups = 0, rsv_gdb = 1024
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
122355712 inodes, 244683776 blocks
12234188 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=247463936
7468 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632,
2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000, 214990848
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [linux-lvm] large logical volumes
2005-03-31 0:29 [linux-lvm] large logical volumes Guido Vettoretti
@ 2005-04-29 20:00 ` Myrddin Emrys
2005-04-29 20:35 ` Guido Vettoretti
0 siblings, 1 reply; 5+ messages in thread
From: Myrddin Emrys @ 2005-04-29 20:00 UTC (permalink / raw)
To: LVM general discussion and development
Looks like you never got a response. I hardly consider myself
qualified, but have you considered increasing the block size? The
other option, have you tried a different filesystem? Different fs have
different limitations... what you're describing appears to be a
limitation of ext2fs, not LVM. Have you considered XFS or ReiserFS?
On 3/30/05, Guido Vettoretti <g.vettoretti@utoronto.ca> wrote:
> Hello,
>
> I'm trying to set up a large logical volume (13.1 TB) on a RHEL 4.0
> server. I have 8 scsi devices of 1.8Tb each.
>
> When I try to do a 'mkfs' and set up a large disk I get limited to 4Tbytes.
> This command creates a smaller filesystem then
> [root@cyclone ~]# lvcreate -l 1287525 -i 3 -n lvol0 vol0
> this one.
> [root@cyclone ~]# lvcreate -l 1000000 -i 3 -n lvol0 vol0
>
> It seems that there is a limit on the number of block groups of 32768
> and blocks per group of 32768.
> My block size is 4096
> so I can only get an LV of 4096*32768*32768 = 4TB.
>
> I tried modifying the PE size on the volume group, but I'm still having
> problems.
>
> Can anyone help please?
>
> Thanks
> Guido
> ---------------
> [root@cyclone ~]# mkfs -t ext3 /dev/vol0/lvol0
> mke2fs 1.35 (28-Feb-2004)
> max_blocks 4294967295, rsv_groups = 0, rsv_gdb = 1024
> Filesystem label=
> OS type: Linux
> Block size=4096 (log=2)
> Fragment size=4096 (log=2)
> 512016384 inodes, 1024002048 blocks
> 8250429 blocks (0.81%) reserved for the super user
> First data block=0
> Maximum filesystem blocks=1027604480
> 31251 block groups
> 32768 blocks per group, 32768 fragments per group
> 16384 inodes per group
> Superblock backups stored on blocks:
> 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632,
> 2654208,
> 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
> 102400000, 214990848, 512000000, 550731776, 644972544
>
> Here is the entire process I typed:
> ------------------------------------------------------------------------------
>
> [root@cyclone ~]# pvcreate /dev/sdc1 /dev/sdd1 /dev/sde1
> Physical volume "/dev/sdc1" successfully created
> Physical volume "/dev/sdd1" successfully created
> Physical volume "/dev/sde1" successfully created
> [root@cyclone ~]# pvscan
> /dev/cdrom: open failed: No medium found
> PV /dev/sda2 VG VolGroup00 lvm2 [34.06 GB / 0 free]
> PV /dev/sdb1 VG VolGroup00 lvm2 [34.16 GB / 0 free]
> PV /dev/sdc1 lvm2 [1.64 TB]
> PV /dev/sdd1 lvm2 [1.64 TB]
> PV /dev/sde1 lvm2 [1.64 TB]
> Total: 5 [4.98 TB] / in use: 2 [68.22 GB] / in no VG: 3 [4.91 TB]
> [root@cyclone ~]# vgcreate vol0 /dev/sd[c-e]1
> Volume group "vol0" successfully created
> [root@cyclone ~]# vgdisplay
> --- Volume group ---
> VG Name vol0
> System ID
> Format lvm2
> Metadata Areas 3
> Metadata Sequence No 1
> VG Access read/write
> VG Status resizable
> MAX LV 0
> Cur LV 0
> Open LV 0
> Max PV 0
> Cur PV 3
> Act PV 3
> VG Size 4.91 TB
> PE Size 4.00 MB
> Total PE 1287525
> Alloc PE / Size 0 / 0
> Free PE / Size 1287525 / 4.91 TB
> VG UUID fQ7cKh-FVnl-NglQ-jimv-RsZK-dU5j-zYKIcG
>
> [root@cyclone ~]# lvcreate -l 1287525 -i 3 -n lvol0 vol0
> Using default stripesize 64KB
> Logical volume "lvol0" created
> [root@cyclone ~]# lvdisplay
> --- Logical volume ---
> LV Name /dev/vol0/lvol0
> VG Name vol0
> LV UUID c2MADb-BoAO-lhl3-3ukj-XfyQ-K8L0-cxcIlw
> LV Write Access read/write
> LV Status available
> # open 0
> LV Size 4.91 TB
> Current LE 1287525
> Segments 1
> Allocation inherit
> Read ahead sectors 0
> Block device 253:2
>
> [root@cyclone ~]# mkfs -t ext3 /dev/vol0/lvol0
> mke2fs 1.35 (28-Feb-2004)
> max_blocks 4294967295, rsv_groups = 0, rsv_gdb = 1024
> Filesystem label=
> OS type: Linux
> Block size=4096 (log=2)
> Fragment size=4096 (log=2)
> 122355712 inodes, 244683776 blocks
> 12234188 blocks (5.00%) reserved for the super user
> First data block=0
> Maximum filesystem blocks=247463936
> 7468 block groups
> 32768 blocks per group, 32768 fragments per group
> 16384 inodes per group
> Superblock backups stored on blocks:
> 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632,
> 2654208,
> 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
> 102400000, 214990848
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [linux-lvm] large logical volumes
2005-04-29 20:00 ` Myrddin Emrys
@ 2005-04-29 20:35 ` Guido Vettoretti
2005-05-03 15:21 ` Dan Pritts
0 siblings, 1 reply; 5+ messages in thread
From: Guido Vettoretti @ 2005-04-29 20:35 UTC (permalink / raw)
To: Myrddin Emrys, LVM general discussion and development
[-- Attachment #1: Type: text/plain, Size: 6190 bytes --]
Someone answered my reponse! Thanks
Anyway. I've sorted things out. I've manage to set up a fairly large
RAID fs on and intel xeon 32-bit server running RHEL4.
As you suggest the solution was to go to a new fs (I chose XFS as Reiser
seemed to behave strangely with LVM and the large filesystem). I just
had to rebuild the kernel with xfs support. Things are running smoothly
right now.
In case anyone is interested there seem to be some limitations:
32-bit machines:
ext3 : 2Tb (I managed to create a 4TB fs for some reason)
xfs : 16 Tb
reiser : 16Tb
64-bit:
ext3: 2Tb (? not sure about this one)
xfs and reiser: very large (> million Tb)
One of the hardware support people said there was a 2Tb limit on the
SCSI protocol (not sure about this), so I had to use LVM to create the
large volume from a subset of RAID volumes even though the native
capacity of the RAID unit was 16Tb. So the process was to divide and
recollect the RAID volumes using LVM and xfs.
Cheers,
Guido
Myrddin Emrys wrote:
>Looks like you never got a response. I hardly consider myself
>qualified, but have you considered increasing the block size? The
>other option, have you tried a different filesystem? Different fs have
>different limitations... what you're describing appears to be a
>limitation of ext2fs, not LVM. Have you considered XFS or ReiserFS?
>
>On 3/30/05, Guido Vettoretti <g.vettoretti@utoronto.ca> wrote:
>
>
>>Hello,
>>
>>I'm trying to set up a large logical volume (13.1 TB) on a RHEL 4.0
>>server. I have 8 scsi devices of 1.8Tb each.
>>
>>When I try to do a 'mkfs' and set up a large disk I get limited to 4Tbytes.
>>This command creates a smaller filesystem then
>>[root@cyclone ~]# lvcreate -l 1287525 -i 3 -n lvol0 vol0
>>this one.
>>[root@cyclone ~]# lvcreate -l 1000000 -i 3 -n lvol0 vol0
>>
>>It seems that there is a limit on the number of block groups of 32768
>>and blocks per group of 32768.
>>My block size is 4096
>>so I can only get an LV of 4096*32768*32768 = 4TB.
>>
>>I tried modifying the PE size on the volume group, but I'm still having
>>problems.
>>
>>Can anyone help please?
>>
>>Thanks
>>Guido
>>---------------
>>[root@cyclone ~]# mkfs -t ext3 /dev/vol0/lvol0
>>mke2fs 1.35 (28-Feb-2004)
>>max_blocks 4294967295, rsv_groups = 0, rsv_gdb = 1024
>>Filesystem label=
>>OS type: Linux
>>Block size=4096 (log=2)
>>Fragment size=4096 (log=2)
>>512016384 inodes, 1024002048 blocks
>>8250429 blocks (0.81%) reserved for the super user
>>First data block=0
>>Maximum filesystem blocks=1027604480
>>31251 block groups
>>32768 blocks per group, 32768 fragments per group
>>16384 inodes per group
>>Superblock backups stored on blocks:
>> 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632,
>>2654208,
>> 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
>> 102400000, 214990848, 512000000, 550731776, 644972544
>>
>>Here is the entire process I typed:
>>------------------------------------------------------------------------------
>>
>>[root@cyclone ~]# pvcreate /dev/sdc1 /dev/sdd1 /dev/sde1
>> Physical volume "/dev/sdc1" successfully created
>> Physical volume "/dev/sdd1" successfully created
>> Physical volume "/dev/sde1" successfully created
>>[root@cyclone ~]# pvscan
>> /dev/cdrom: open failed: No medium found
>> PV /dev/sda2 VG VolGroup00 lvm2 [34.06 GB / 0 free]
>> PV /dev/sdb1 VG VolGroup00 lvm2 [34.16 GB / 0 free]
>> PV /dev/sdc1 lvm2 [1.64 TB]
>> PV /dev/sdd1 lvm2 [1.64 TB]
>> PV /dev/sde1 lvm2 [1.64 TB]
>> Total: 5 [4.98 TB] / in use: 2 [68.22 GB] / in no VG: 3 [4.91 TB]
>>[root@cyclone ~]# vgcreate vol0 /dev/sd[c-e]1
>> Volume group "vol0" successfully created
>>[root@cyclone ~]# vgdisplay
>> --- Volume group ---
>> VG Name vol0
>> System ID
>> Format lvm2
>> Metadata Areas 3
>> Metadata Sequence No 1
>> VG Access read/write
>> VG Status resizable
>> MAX LV 0
>> Cur LV 0
>> Open LV 0
>> Max PV 0
>> Cur PV 3
>> Act PV 3
>> VG Size 4.91 TB
>> PE Size 4.00 MB
>> Total PE 1287525
>> Alloc PE / Size 0 / 0
>> Free PE / Size 1287525 / 4.91 TB
>> VG UUID fQ7cKh-FVnl-NglQ-jimv-RsZK-dU5j-zYKIcG
>>
>>[root@cyclone ~]# lvcreate -l 1287525 -i 3 -n lvol0 vol0
>> Using default stripesize 64KB
>> Logical volume "lvol0" created
>>[root@cyclone ~]# lvdisplay
>> --- Logical volume ---
>> LV Name /dev/vol0/lvol0
>> VG Name vol0
>> LV UUID c2MADb-BoAO-lhl3-3ukj-XfyQ-K8L0-cxcIlw
>> LV Write Access read/write
>> LV Status available
>> # open 0
>> LV Size 4.91 TB
>> Current LE 1287525
>> Segments 1
>> Allocation inherit
>> Read ahead sectors 0
>> Block device 253:2
>>
>>[root@cyclone ~]# mkfs -t ext3 /dev/vol0/lvol0
>>mke2fs 1.35 (28-Feb-2004)
>>max_blocks 4294967295, rsv_groups = 0, rsv_gdb = 1024
>>Filesystem label=
>>OS type: Linux
>>Block size=4096 (log=2)
>>Fragment size=4096 (log=2)
>>122355712 inodes, 244683776 blocks
>>12234188 blocks (5.00%) reserved for the super user
>>First data block=0
>>Maximum filesystem blocks=247463936
>>7468 block groups
>>32768 blocks per group, 32768 fragments per group
>>16384 inodes per group
>>Superblock backups stored on blocks:
>> 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632,
>>2654208,
>> 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
>> 102400000, 214990848
>>
>>_______________________________________________
>>linux-lvm mailing list
>>linux-lvm@redhat.com
>>https://www.redhat.com/mailman/listinfo/linux-lvm
>>read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>>
>>
>>
>
>_______________________________________________
>linux-lvm mailing list
>linux-lvm@redhat.com
>https://www.redhat.com/mailman/listinfo/linux-lvm
>read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>
>
>
>
[-- Attachment #2: Type: text/html, Size: 7016 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [linux-lvm] large logical volumes
2005-04-29 20:35 ` Guido Vettoretti
@ 2005-05-03 15:21 ` Dan Pritts
2005-05-03 16:42 ` Guido Vettoretti
0 siblings, 1 reply; 5+ messages in thread
From: Dan Pritts @ 2005-05-03 15:21 UTC (permalink / raw)
To: LVM general discussion and development
On Fri, Apr 29, 2005 at 04:35:02PM -0400, Guido Vettoretti wrote:
> 32-bit machines:
> ext3 : 2Tb (I managed to create a 4TB fs for some reason)
I'm not sure this is true - i think that this limit was there but was
removed in newer versions of ext2/3
> ext3: 2Tb (? not sure about this one)
I have created larger filesystems on 64-bit.
> One of the hardware support people said there was a 2Tb limit on the
> SCSI protocol (not sure about this),
More or less true, but some newer hardware and device drivers
have fixed this. The keyword you need to look for is "64-bit LBA".
I've had success with an Atto celerity fibre channel adapter and
an infortrend-based RAID on x86_64. However, each one needed a
firmware/driver update from what was shipped to me in December of
last year to work.
tnx
danno
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [linux-lvm] large logical volumes
2005-05-03 15:21 ` Dan Pritts
@ 2005-05-03 16:42 ` Guido Vettoretti
0 siblings, 0 replies; 5+ messages in thread
From: Guido Vettoretti @ 2005-05-03 16:42 UTC (permalink / raw)
To: LVM general discussion and development
[-- Attachment #1: Type: text/plain, Size: 1644 bytes --]
Dan Pritts wrote:
>On Fri, Apr 29, 2005 at 04:35:02PM -0400, Guido Vettoretti wrote:
>
>
>>32-bit machines:
>>ext3 : 2Tb (I managed to create a 4TB fs for some reason)
>>
>>
>
>
>
>I'm not sure this is true - i think that this limit was there but was
>removed in newer versions of ext2/3
>
>
>
I haven't heard anything about the new versions but I was workiing with
kernel 2.6.9, and the number of block groups would roll over to 0 when I
specified a number greater than 32768. Thus the max limitation seemes to
be 4096(bytes/block)*32768(blocks/group)*32768(blockgroups) = 4TB
[root@cyclone ~]# mkfs -t ext3 /dev/vol0/lvol0
mke2fs 1.35 (28-Feb-2004)
max_blocks 4294967295, rsv_groups = 0, rsv_gdb = 1024
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
512016384 inodes, 1024002048 blocks
8250429 blocks (0.81%) reserved for the super user
First data block=0
Maximum filesystem blocks=1027604480
31251 block groups
32768 blocks per group, 32768 fragments per group
>>ext3: 2Tb (? not sure about this one)
>>
>>
>
>I have created larger filesystems on 64-bit.
>
>
>
>>One of the hardware support people said there was a 2Tb limit on the
>>SCSI protocol (not sure about this),
>>
>>
>
>More or less true, but some newer hardware and device drivers
>have fixed this. The keyword you need to look for is "64-bit LBA".
>
>I've had success with an Atto celerity fibre channel adapter and
>an infortrend-based RAID on x86_64. However, each one needed a
>firmware/driver update from what was shipped to me in December of
>last year to work.
>
>
>
This is useful info,
Thanks
Guido
[-- Attachment #2: Type: text/html, Size: 2468 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2005-05-03 16:42 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2005-03-31 0:29 [linux-lvm] large logical volumes Guido Vettoretti
2005-04-29 20:00 ` Myrddin Emrys
2005-04-29 20:35 ` Guido Vettoretti
2005-05-03 15:21 ` Dan Pritts
2005-05-03 16:42 ` Guido Vettoretti
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.