How to resize to an bigger then 16TB ext4 filesysteem on Debian

How to resize to an bigger then 16TB ext4 filesysteem on Debian

[Jelle de Jong]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello everyone,

https://packages.debian.org/wheezy/e2fsprogs

# resize2fs /dev/lvm0-vol/storage
resize2fs 1.42.5 (29-Jul-2012)
resize2fs: New size too large to be expressed in 32 bits

(will the command work with e2fsprogs 1.42.9-3?)

# tune2fs -l /dev/lvm0-vol/storage
http://paste.debian.net/92889/

Suggestions has to be stable enough for production systems.

I now did a lvreduce -L 16TB /dev/lvm0-vol/storage and resize2fs
/dev/lvm0-vol/storage (its running now)

# resize2fs /dev/lvm0-vol/storage
resize2fs 1.42.5 (29-Jul-2012)
Filesystem at /dev/lvm0-vol/storage is mounted on /srv/storage;
on-line resizing required
old_desc_blocks = 464, new_desc_blocks = 1024
Performing an on-line resize of /dev/lvm0-vol/storage to 4294967295
(4k) blocks.

Kind regards,

Jelle de Jong
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iJwEAQECAAYFAlNGgFAACgkQ1WclBW9j5HnDbQP/YRQvZq0HPJ4brpBas+95xeP3
lOnN0OzEOEpjC9pY7J9ppWfkvOkBx7ayjt8MqmJgluwVBNQIRUU3ctftaI7cUPTH
abgj0JHLICCjSIs3nVzQo9pq8URnhTbDXDcXLBemglT5wa+Potlz4YYFA+9Cd/ty
G2jfKBiaNCN17Nmikjo=
=bwbX
-----END PGP SIGNATURE-----

Re: How to resize to an bigger then 16TB ext4 filesysteem on Debian

[Theodore Ts'o]

On Thu, Apr 10, 2014 at 01:28:21PM +0200, Jelle de Jong wrote:
> 
> Hello everyone,
> 
> https://packages.debian.org/wheezy/e2fsprogs
> 
> # resize2fs /dev/lvm0-vol/storage
> resize2fs 1.42.5 (29-Jul-2012)
> resize2fs: New size too large to be expressed in 32 bits
> 
> (will the command work with e2fsprogs 1.42.9-3?)
> 
> # tune2fs -l /dev/lvm0-vol/storage
> http://paste.debian.net/92889/
> 
> Suggestions has to be stable enough for production systems.

I'm going to assume the file system was created without having the
64-bit file system feature set.  (Running dumpe2fs -h
/dev/lvm0-vol/storage can verify this).

If that's true, and you require a production-stable way of doing this,
there's really only one thing I can suggest:

1)  Backup your file system completely
2)  Get the development version of e2fsprogs, and apply a set of
    in-development patches that haven't yet been merged
3)  Use this in-development resize2fs to convert to a 64-bit file system,
    realizing that it will have completely rewrite your inode tables.
    Cross your fingers.
4)  If it works, you're done.  If not, you can restore from backups.  :-)

So basically steps 2 and 3 are ways of optimizing needing to do a full
restore of your file system.....

Cheers,

					- Ted

Re: How to resize to an bigger then 16TB ext4 filesysteem on Debian

[Jelle de Jong]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 10/04/14 16:09, Theodore Ts'o wrote:
> On Thu, Apr 10, 2014 at 01:28:21PM +0200, Jelle de Jong wrote:
>> 
>> Hello everyone,
>> 
>> https://packages.debian.org/wheezy/e2fsprogs
>> 
>> # resize2fs /dev/lvm0-vol/storage resize2fs 1.42.5 (29-Jul-2012) 
>> resize2fs: New size too large to be expressed in 32 bits
>> 
>> (will the command work with e2fsprogs 1.42.9-3?)
>> 
>> # tune2fs -l /dev/lvm0-vol/storage 
>> http://paste.debian.net/92889/
>> 
>> Suggestions has to be stable enough for production systems.
> 
> I'm going to assume the file system was created without having the 
> 64-bit file system feature set.  (Running dumpe2fs -h 
> /dev/lvm0-vol/storage can verify this).
> 
> If that's true, and you require a production-stable way of doing
> this, there's really only one thing I can suggest:
> 
> 1)  Backup your file system completely 2)  Get the development
> version of e2fsprogs, and apply a set of in-development patches
> that haven't yet been merged 3)  Use this in-development resize2fs
> to convert to a 64-bit file system, realizing that it will have
> completely rewrite your inode tables. Cross your fingers. 4)  If it
> works, you're done.  If not, you can restore from backups.  :-)
> 
> So basically steps 2 and 3 are ways of optimizing needing to do a
> full restore of your file system.....

Thank you Ted for your quick reply.

In october 2012 I created the ext4 file system with the command:
mkfs.ext4 -m 1 /dev/lvm0-vol/storage

I assumed by using ext4 with lvm2 I would have had a solid basis for
some future storage expeditions. Apparently I made a mistake somewhere
but I don't know where yet or did I hit a bug?

# tune2fs -l /dev/lvm0-vol/storage
# dumpe2fs -h /dev/lvm0-vol/storage
http://paste.debian.net/plain/92953

Where can I see that my ext4 file system is limed to 16TB?

Your suggestion to use a development version of e2fsprogs doesn't
really suit my needs for a stable solution that is maintainable.

1. I can temporary transfer all the data to an other system and
recreate the ext4 file system, but will this be stable with version
1.42.5 and how much TB can the file-system handle?

2. I can put current the 16TB ext4 file system in production and wait
for e2fsprogs support in Debian stable or backports to arrive and
resize the to 22TB later. When do you expect this functionality to
arrive in as stable release of e2fsprogs?

Thank you for your help.

Kind regards,

Jelle de Jong
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iJwEAQECAAYFAlNGs7IACgkQ1WclBW9j5Hl5NgQAjwMLRPu3KQuBesD7BSHkUrf3
JcaFMD90vvAYmbKMbMumGhR86BA1NXVBfeEA6DnrZUrPzly1VBglCOwCJzw52GCc
TZHp3h5RD/Ce1c8/T+VD4DrVxhjIMUAyUT+/eGN008mHtrOfRppnFlcutNxZ5RZJ
oGksAsJ5i7hWmeilqcc=
=ECQP
-----END PGP SIGNATURE-----

Re: How to resize to an bigger then 16TB ext4 filesysteem on Debian

[Theodore Ts'o]

On Thu, Apr 10, 2014 at 05:07:34PM +0200, Jelle de Jong wrote:
> 
> In october 2012 I created the ext4 file system with the command:
> mkfs.ext4 -m 1 /dev/lvm0-vol/storage
> 
> I assumed by using ext4 with lvm2 I would have had a solid basis for
> some future storage expeditions. Apparently I made a mistake somewhere
> but I don't know where yet or did I hit a bug?
> 
> # tune2fs -l /dev/lvm0-vol/storage
> # dumpe2fs -h /dev/lvm0-vol/storage
> http://paste.debian.net/plain/92953
> 
> Where can I see that my ext4 file system is limed to 16TB?

So here's the problem.  The original ext2/3 block group descriptors
only had room for 32-bit block numbers.  So in order to resize the
file system past 16TB, it was necessary to change the layout of the
block group descriptors, and change them from needing 32 bytes to 64
bytes per block group.

So in order to make this change there was a new "64-bit" feature flag
that changes the size of the block group descriptors.  We didn't
enable this feature at first, because we wanted to make sure the
64-bit support was well shaken down and "production ready" before it
was enabled.

(In particular there were some online resizing bugs relating to 64-bit
file systems that wasn't ironed out until relatively recently, so you
should be glad we didn't enable this feature by default.  :-)

> Your suggestion to use a development version of e2fsprogs doesn't
> really suit my needs for a stable solution that is maintainable.

Well, unfortunately, there is no way to do what you request using the
stable version of e2fsprogs.  Sorry.

> 1. I can temporary transfer all the data to an other system and
> recreate the ext4 file system, but will this be stable with version
> 1.42.5 and how much TB can the file-system handle?

You should definitely go to the 1.42.9 version of e2fsprogs to make
sure you have the latest fixes.

As far as the kernel is concerned, are you still using the Debian
stable 3.2 kernel?  The problem is that not all of the bug fixes get
backported to a stable kernel which is that old.  One of the value
adds that a company like Red Hat provides is that they have someone
paid to try to support ext4 on older kernels.  But there have been
some online resize fixes on the kernel side since 3.2, and I don't
think all of them have been backported because they are fairly complex
and involved patches.

If you're not willing to use the kernel from Debian testing, it may
very well be that you will be better off waiting for Debian testing to
ship before you even try to use ext4 file systems > that 16TB.  I
really can't guarantee that all of the necessary fixes have been
backported to the 3.2 stable kernel.

Is there some way you could live with two large file systems instead
of trying to create one very large file system?

Regards,

						- Ted