Problem with 15TB logical volume

Problem with 15TB logical volume

[Adam Huffman]

Hope it's okay to send "user problems" here.  

I've made a 15TB logical volume on a 21TB RAID6 iSCSI array.

When I try to make an ext4 filesystem I see the following error:

mke4fs -E stride=32 stripe-width=384 -i 65536 -j -v \
/dev/mapper/vg2md3000isata-homes2 

mke4fs 1.41.9 (22-Aug-2009)
mke4fs: invalid blocks count - /dev/mapper/vg2md3000isata-homes2

Some more details:

--- Physical volume ---
  PV Name               /dev/mpath/mpath4
  VG Name               vg2md3000isata
  PV Size               21.83 TB / not usable 32.00 MB
  Allocatable           yes 
  PE Size (KByte)       32768
  Total PE              715205
  Free PE               223685
  Allocated PE          491520
  PV UUID               psv5BF-Tkvv-p7uq-ED0k-DCPd-3es1-uNnNSg

--- Logical volume ---
  LV Name                /dev/vg2md3000isata/homes2
  VG Name                vg2md3000isata
  LV UUID                qieiLw-F0he-DlDD-bZeA-v2Ok-DmE5-oS3uy2
  LV Write Access        read/write
  LV Status              available
  # open                 0
  LV Size                15.00 TB
  Current LE             491520
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:11

--- Volume group ---
  VG Name               vg2md3000isata
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  2
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               21.83 TB
  PE Size               32.00 MB
  Total PE              715205
  Alloc PE / Size       491520 / 15.00 TB
  Free  PE / Size       223685 / 6.83 TB
  VG UUID               vLwJLm-z1gj-RZrM-GjXX-hmLo-cf6B-8EGMyR

When i searched briefly yesterday, some documentation stated that the
filesystem limit was now 16TB, and some that the limit was much higher.

The system is running Centos 5.5, e4fsprogs-1.41.9-3.el5.


Adam

Re: Problem with 15TB logical volume

[Eric Sandeen]

Adam Huffman wrote:
> Hope it's okay to send "user problems" here.  

Sure, that's fine.

> I've made a 15TB logical volume on a 21TB RAID6 iSCSI array.
> 
> When I try to make an ext4 filesystem I see the following error:
> 
> mke4fs -E stride=32 stripe-width=384 -i 65536 -j -v \
> /dev/mapper/vg2md3000isata-homes2 
> 
> mke4fs 1.41.9 (22-Aug-2009)
> mke4fs: invalid blocks count - /dev/mapper/vg2md3000isata-homes2


You're not specifying the -E options correctly:

[root@host e2fsprogs]# misc/mke2fs.static -t ext4 -E stride=32 stripe-width=384 /mnt/test2/testdir/bigfile 
mke2fs 1.41.12 (17-May-2010)
mke2fs.static: invalid blocks count - /mnt/test2/testdir/bigfile

This works - you need comma-separated -E options:

[root@host e2fsprogs]# mke4fs -E stride=32,stripe-width=384 /mnt/test2/testdir/bigfile 
mke4fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=32 blocks, Stripe width=384 blocks
1006632960 inodes, 4026531840 blocks
...

Without the comma, mkfs thinks you are trying to mkfs a device called
"stripe-width=384" with a block count of "<devicename>"

Odd eh ;)

Perhaps echoing back the blocks count which it found to be invalid
(in this case the devicename string) would make the failure more
obvious...


> When i searched briefly yesterday, some documentation stated that the
> filesystem limit was now 16TB, and some that the limit was much higher.

Yep that should work.

-Eric
 
> The system is running Centos 5.5, e4fsprogs-1.41.9-3.el5.

Re: Problem with 15TB logical volume

[Eric Sandeen]

Eric Sandeen wrote:

...

> Without the comma, mkfs thinks you are trying to mkfs a device called
> "stripe-width=384" with a block count of "<devicename>"
> 
> Odd eh ;)
> 
> Perhaps echoing back the blocks count which it found to be invalid
> (in this case the devicename string) would make the failure more
> obvious...
> 

Oh, actually, that is just what it's doing, but it's not obvious,
since it was the device name, and it looked like it was simply reporting
which device had the error...

-Eric

Re: Problem with 15TB logical volume

[Adam Huffman]

On Thu, Aug 12, 2010 at 09:43:29AM -0400, Eric Sandeen wrote:
> 
> > I've made a 15TB logical volume on a 21TB RAID6 iSCSI array.
> > 
> > When I try to make an ext4 filesystem I see the following error:
> > 
> > mke4fs -E stride=32 stripe-width=384 -i 65536 -j -v \
> > /dev/mapper/vg2md3000isata-homes2 
> > 
> > mke4fs 1.41.9 (22-Aug-2009)
> > mke4fs: invalid blocks count - /dev/mapper/vg2md3000isata-homes2
> 
> 
> You're not specifying the -E options correctly:
> 
> [root@host e2fsprogs]# misc/mke2fs.static -t ext4 -E stride=32 stripe-width=384 /mnt/test2/testdir/bigfile 
> mke2fs 1.41.12 (17-May-2010)
> mke2fs.static: invalid blocks count - /mnt/test2/testdir/bigfile
> 
> This works - you need comma-separated -E options:
> 
> [root@host e2fsprogs]# mke4fs -E stride=32,stripe-width=384 /mnt/test2/testdir/bigfile 
> mke4fs 1.41.12 (17-May-2010)
> Filesystem label=
> OS type: Linux
> Block size=4096 (log=2)
> Fragment size=4096 (log=2)
> Stride=32 blocks, Stripe width=384 blocks
> 1006632960 inodes, 4026531840 blocks
> ...
> 
> Without the comma, mkfs thinks you are trying to mkfs a device called
> "stripe-width=384" with a block count of "<devicename>"
> 
> Odd eh ;)
> 

Yes, thanks for that.  It's happy now.

> Perhaps echoing back the blocks count which it found to be invalid
> (in this case the devicename string) would make the failure more
> obvious...
> 

Perhaps it could check for a clearly non-numeric input and report that?
Or maybe it's just me who'd do such a thing...


> 
> > When i searched briefly yesterday, some documentation stated that the
> > filesystem limit was now 16TB, and some that the limit was much higher.
> 
> Yep that should work.
> 

Looking at
https://bugs.launchpad.net/ubuntu/+source/e2fsprogs/+bug/521648,
specifically Ted's comment, is it correct that 64-bit blocks support is
still at the early development stage?

Adam


> -Eric
>  
> > The system is running Centos 5.5, e4fsprogs-1.41.9-3.el5.
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: Problem with 15TB logical volume

[Eric Sandeen]

Adam Huffman wrote:

> Looking at
> https://bugs.launchpad.net/ubuntu/+source/e2fsprogs/+bug/521648,
> specifically Ted's comment, is it correct that 64-bit blocks support is
> still at the early development stage?
> 
> Adam


the code is in a branch in ted's e2fsprogs git tree but not yet in a release.

-Eric

Re: Problem with 15TB logical volume

[Andreas Dilger]

On 2010-08-12, at 08:51, Adam Huffman wrote:
>>> mke4fs -E stride=32 stripe-width=384 -i 65536 -j -v \
>>> /dev/mapper/vg2md3000isata-homes2 
>>> 
>>> mke4fs 1.41.9 (22-Aug-2009)
>>> mke4fs: invalid blocks count - /dev/mapper/vg2md3000isata-homes2

I think if this was reported as:

mke4fs: invalid blocks count '/dev/mapper/vg2md3000isata-homes2'

it would definitely help.

Cheers, Andreas

Re: Problem with 15TB logical volume

[Eric Sandeen]

Andreas Dilger wrote:
> On 2010-08-12, at 08:51, Adam Huffman wrote:
>>>> mke4fs -E stride=32 stripe-width=384 -i 65536 -j -v \
>>>> /dev/mapper/vg2md3000isata-homes2 
>>>>
>>>> mke4fs 1.41.9 (22-Aug-2009)
>>>> mke4fs: invalid blocks count - /dev/mapper/vg2md3000isata-homes2
> 
> I think if this was reported as:
> 
> mke4fs: invalid blocks count '/dev/mapper/vg2md3000isata-homes2'
> 
> it would definitely help.


maybe even better:

mke4fs: invalid blocks count '/dev/mapper/vg2md3000isata-homes2' on device 'stripe-width=384'

;)

-Eric