EXT4 Larger Than 16TB

EXT4 Larger Than 16TB

[Michael]

I have a currently 15TB FS that I want to expand to 18TB.
Unfortunately EXT4 e2fsprogs has an issue that stops you from being
able to do this:

sudo resize2fs /dev/md0
resize2fs 1.42.5 (29-Jul-2012)
resize2fs: New size too large to be expressed in 32 bits

It seems to me, reading the change logs, that this only supported if
you created your FS with 64 bit specified. There seems to have been a
patch last year that added support for changing 32->64bit, but I had a
hard time finding/tracking that.

What is the state of expanding an ext4 FS larger than 16tb?

I am not subscribed, please include me in reply.

Thank you!

Re: EXT4 Larger Than 16TB

[Theodore Ts'o]

On Mon, Jul 08, 2013 at 04:32:50PM -0500, Michael wrote:
> I have a currently 15TB FS that I want to expand to 18TB.
> Unfortunately EXT4 e2fsprogs has an issue that stops you from being
> able to do this:
> 
> sudo resize2fs /dev/md0
> resize2fs 1.42.5 (29-Jul-2012)
> resize2fs: New size too large to be expressed in 32 bits
> 
> It seems to me, reading the change logs, that this only supported if
> you created your FS with 64 bit specified. There seems to have been a
> patch last year that added support for changing 32->64bit, but I had a
> hard time finding/tracking that.

Unfortuantely, there is no patch that would enable this.

In theory it could be done; it would require doubling the size of the
block group descriptors, which would require moving other metadata
blocks out of the way.  Resize2fs does do this when growing a file
system off-line when no blocks are available from the resize inode,
but no one has implemented the necessary changes to extend this to
adding the 64-bit feature.

So it's technically possible, but it's not implemented at this time.

Sorry,

							- Ted

Re: EXT4 Larger Than 16TB

[Michael]

No problem, it sounds like it would be a very tricky bit of code to
write. Shuffling metadata blocks is not an easy feat.

Is there any way to tell if I'm using 64 bit? Ubuntu defaults to
setting ext4 to default, but I don't know if it did when I first
created this FS.

If it is 64 bit, the resize2fs utility should be able to handle this
one kernels >3.7, correct?

On Mon, Jul 8, 2013 at 9:14 PM, Theodore Ts'o <tytso@mit.edu> wrote:
> On Mon, Jul 08, 2013 at 04:32:50PM -0500, Michael wrote:
>> I have a currently 15TB FS that I want to expand to 18TB.
>> Unfortunately EXT4 e2fsprogs has an issue that stops you from being
>> able to do this:
>>
>> sudo resize2fs /dev/md0
>> resize2fs 1.42.5 (29-Jul-2012)
>> resize2fs: New size too large to be expressed in 32 bits
>>
>> It seems to me, reading the change logs, that this only supported if
>> you created your FS with 64 bit specified. There seems to have been a
>> patch last year that added support for changing 32->64bit, but I had a
>> hard time finding/tracking that.
>
> Unfortuantely, there is no patch that would enable this.
>
> In theory it could be done; it would require doubling the size of the
> block group descriptors, which would require moving other metadata
> blocks out of the way.  Resize2fs does do this when growing a file
> system off-line when no blocks are available from the resize inode,
> but no one has implemented the necessary changes to extend this to
> adding the 64-bit feature.
>
> So it's technically possible, but it's not implemented at this time.
>
> Sorry,
>
>                                                         - Ted

Re: EXT4 Larger Than 16TB

[H. Peter Anvin]

On 07/09/2013 06:45 AM, Michael wrote:
> No problem, it sounds like it would be a very tricky bit of code to
> write. Shuffling metadata blocks is not an easy feat.
> 
> Is there any way to tell if I'm using 64 bit? Ubuntu defaults to
> setting ext4 to default, but I don't know if it did when I first
> created this FS.
> 
> If it is 64 bit, the resize2fs utility should be able to handle this
> one kernels >3.7, correct?
> 

Is there a way to force this at mkfs time?  I.e. "I know I may want to
expand this filesystem beyond the 16 TB point?"

	-hpa

Re: EXT4 Larger Than 16TB

[Zheng Liu]

On Sun, Jul 21, 2013 at 03:47:04PM -0700, H. Peter Anvin wrote:
> On 07/09/2013 06:45 AM, Michael wrote:
> > No problem, it sounds like it would be a very tricky bit of code to
> > write. Shuffling metadata blocks is not an easy feat.
> > 
> > Is there any way to tell if I'm using 64 bit? Ubuntu defaults to
> > setting ext4 to default, but I don't know if it did when I first
> > created this FS.
> > 
> > If it is 64 bit, the resize2fs utility should be able to handle this
> > one kernels >3.7, correct?
> > 
> 
> Is there a way to force this at mkfs time?  I.e. "I know I may want to
> expand this filesystem beyond the 16 TB point?"

Hi Peter,

You could use 'mkfs.ext4 -O 64bit' to force enable 64bit support.  In
addition, we will enable this feature by default.  Lukas has a patch to
fix it.  Here is the link [1].

1. http://patchwork.ozlabs.org/patch/253234/

Regards,
                                                - Zheng

Re: EXT4 Larger Than 16TB

[Michael]

Also, in /etc/mke2fs.conf(Ubuntu has this by default):
        ext4 = {
                features =
has_journal,extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize
                auto_64-bit_support = 1
                inode_size = 256
        }

        ext4dev = {
                features =
has_journal,extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize
                inode_size = 256
                options = test_fs=1
        }



Also, any info on how to find out if my fs is 64 bit or not? I don't
know what this config was back when I made the system.

On Sun, Jul 21, 2013 at 6:43 PM, Zheng Liu <gnehzuil.liu@gmail.com> wrote:
> On Sun, Jul 21, 2013 at 03:47:04PM -0700, H. Peter Anvin wrote:
>> On 07/09/2013 06:45 AM, Michael wrote:
>> > No problem, it sounds like it would be a very tricky bit of code to
>> > write. Shuffling metadata blocks is not an easy feat.
>> >
>> > Is there any way to tell if I'm using 64 bit? Ubuntu defaults to
>> > setting ext4 to default, but I don't know if it did when I first
>> > created this FS.
>> >
>> > If it is 64 bit, the resize2fs utility should be able to handle this
>> > one kernels >3.7, correct?
>> >
>>
>> Is there a way to force this at mkfs time?  I.e. "I know I may want to
>> expand this filesystem beyond the 16 TB point?"
>
> Hi Peter,
>
> You could use 'mkfs.ext4 -O 64bit' to force enable 64bit support.  In
> addition, we will enable this feature by default.  Lukas has a patch to
> fix it.  Here is the link [1].
>
> 1. http://patchwork.ozlabs.org/patch/253234/
>
> Regards,
>                                                 - Zheng

Re: EXT4 Larger Than 16TB

[Zheng Liu]

Hi Michael,

On Sun, Jul 21, 2013 at 07:44:12PM -0500, Michael wrote:
> Also, in /etc/mke2fs.conf(Ubuntu has this by default):
>         ext4 = {
>                 features =
> has_journal,extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize
>                 auto_64-bit_support = 1
>                 inode_size = 256
>         }
> 
>         ext4dev = {
>                 features =
> has_journal,extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize
>                 inode_size = 256
>                 options = test_fs=1
>         }
> 
> 
> 
> Also, any info on how to find out if my fs is 64 bit or not? I don't
> know what this config was back when I made the system.

You could use 'tune2fs -l ${DEV}' to check whether 64bit feature is
enabled.  Here is a sample result:

$ sudo tune2fs -l /dev/sda1
tune2fs 1.43-WIP (21-Jan-2013)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          5abc13bd-01d9-41ff-b7ca-a028b3d610f8
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index
filetype extent 64bit flex_bg sparse_super large_file huge_file
                ^^^^^

uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              5021696
Block count:              20081242
Reserved block count:     1004062
Free blocks:              19719931
Free inodes:              5021685
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      1024
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Filesystem created:       Mon Jul 22 08:50:15 2013
Last mount time:          n/a
Last write time:          Mon Jul 22 08:50:25 2013
Mount count:              0
Maximum mount count:      -1
Last checked:             Mon Jul 22 08:50:15 2013
Check interval:           0 (<none>)
Lifetime writes:          1358 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      7d9a465d-3e37-4d41-af45-fff9c682f8ec
Journal backup:           inode blocks

Regards,
                                                - Zheng

Re: EXT4 Larger Than 16TB

[Michael]

Thank you for your help. Unfortunately, it didn't give me the answer I
had hoped for. Darn!

--
tune2fs 1.42.5 (29-Jul-2012)
Filesystem volume name:   <none>
Last mounted on:          /mnt/dataRaid
Filesystem UUID:          60171591-358c-4b6a-9b29-684c1b964aa8
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index
filetype needs_recovery extent flex_bg sparse_super large_file
huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              503316480
Block count:              4026531840
Reserved block count:     201326591
Free blocks:              1301052106
Free inodes:              502075987
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      64
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         4096
Inode blocks per group:   256
RAID stride:              128
RAID stripe width:        512
Flex block group size:    16
Filesystem created:       Fri Dec  7 01:00:01 2012
Last mount time:          Tue Jul 16 21:09:31 2013
Last write time:          Tue Jul 16 21:09:31 2013
Mount count:              2
Maximum mount count:      -1
Last checked:             Mon Jul  8 15:21:15 2013
Check interval:           0 (<none>)
Lifetime writes:          9 TB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      62024c47-022f-448e-a697-e3fb4ed618f3
Journal backup:           inode blocks

On Sun, Jul 21, 2013 at 7:52 PM, Zheng Liu <gnehzuil.liu@gmail.com> wrote:
> Hi Michael,
>
> On Sun, Jul 21, 2013 at 07:44:12PM -0500, Michael wrote:
>> Also, in /etc/mke2fs.conf(Ubuntu has this by default):
>>         ext4 = {
>>                 features =
>> has_journal,extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize
>>                 auto_64-bit_support = 1
>>                 inode_size = 256
>>         }
>>
>>         ext4dev = {
>>                 features =
>> has_journal,extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize
>>                 inode_size = 256
>>                 options = test_fs=1
>>         }
>>
>>
>>
>> Also, any info on how to find out if my fs is 64 bit or not? I don't
>> know what this config was back when I made the system.
>
> You could use 'tune2fs -l ${DEV}' to check whether 64bit feature is
> enabled.  Here is a sample result:
>
> $ sudo tune2fs -l /dev/sda1
> tune2fs 1.43-WIP (21-Jan-2013)
> Filesystem volume name:   <none>
> Last mounted on:          <not available>
> Filesystem UUID:          5abc13bd-01d9-41ff-b7ca-a028b3d610f8
> Filesystem magic number:  0xEF53
> Filesystem revision #:    1 (dynamic)
> Filesystem features:      has_journal ext_attr resize_inode dir_index
> filetype extent 64bit flex_bg sparse_super large_file huge_file
>                 ^^^^^
>
> uninit_bg dir_nlink extra_isize
> Filesystem flags:         signed_directory_hash
> Default mount options:    user_xattr acl
> Filesystem state:         clean
> Errors behavior:          Continue
> Filesystem OS type:       Linux
> Inode count:              5021696
> Block count:              20081242
> Reserved block count:     1004062
> Free blocks:              19719931
> Free inodes:              5021685
> First block:              0
> Block size:               4096
> Fragment size:            4096
> Reserved GDT blocks:      1024
> Blocks per group:         32768
> Fragments per group:      32768
> Inodes per group:         8192
> Inode blocks per group:   512
> Flex block group size:    16
> Filesystem created:       Mon Jul 22 08:50:15 2013
> Last mount time:          n/a
> Last write time:          Mon Jul 22 08:50:25 2013
> Mount count:              0
> Maximum mount count:      -1
> Last checked:             Mon Jul 22 08:50:15 2013
> Check interval:           0 (<none>)
> Lifetime writes:          1358 MB
> Reserved blocks uid:      0 (user root)
> Reserved blocks gid:      0 (group root)
> First inode:              11
> Inode size:               256
> Required extra isize:     28
> Desired extra isize:      28
> Journal inode:            8
> Default directory hash:   half_md4
> Directory Hash Seed:      7d9a465d-3e37-4d41-af45-fff9c682f8ec
> Journal backup:           inode blocks
>
> Regards,
>                                                 - Zheng