16 TB filesystem limit on 32bit machine

16 TB filesystem limit on 32bit machine

[Rabeeh Khoury]

I'm trying to figure out all issues with regards >16TB filesystem
support on ARM (32bit) machines.
Clearly this issue was hot few years ago, part of the discussions -

https://bugzilla.kernel.org/show_bug.cgi?id=12556
http://www.redhat.com/archives/dm-devel/2009-July/msg00131.html

And there was Eric's patch of checking length of pgoff_t and
accordingly refuse mount.

Now, today with 4TB hard drives in the market, having 5 of those on an
ARM machine is really common and the ext4 limitation is becoming more
reachable and requires attention.
What i'm trying to achieve is the following two items -

---- item #1 ---
Understand where the limitation is really coming from? Is this ext4
implementation limitation or 32bit machines will never work with >16TB
filesystems?
I understand that there is a 16TB file size limitation (2^32*4K page
size so you won't be able to mmap() further than that point) but how
is that related to filesystem size?
Will 64KB page size fix this issue (ARM supports 4KB and 64KB pages) -
clearly memory fragmentation will be a hit here.

----- item #2 ---
Reproduce a failing scenario.
For now i'v created a 24TB volume (thin provisioned) - RAID-0 on a 3 x
loopback on a 3 x truncted 8TB consisting total of 24TB volume
mkfs.ext4 /dev/md0 (e2fsprogrs 1.42 - thanks for the >16TB support)
mount on a hacked kernel (#define pgoff_t unsigned long long thus
making filesystem mounting check disappear)
The volume mounts ok; now how do i get into corruption? I don't have
physical 24TB drive, so best if there is a pin-pointed to test to
reproduce the issue.

Best regards,
Rabeeh

Re: 16 TB filesystem limit on 32bit machine

[Eric Sandeen]

On 2/20/12 9:31 AM, Rabeeh Khoury wrote:
> I'm trying to figure out all issues with regards >16TB filesystem
> support on ARM (32bit) machines.
> Clearly this issue was hot few years ago, part of the discussions -
> 
> https://bugzilla.kernel.org/show_bug.cgi?id=12556
> http://www.redhat.com/archives/dm-devel/2009-July/msg00131.html
> 
> And there was Eric's patch of checking length of pgoff_t and
> accordingly refuse mount.
> 
> Now, today with 4TB hard drives in the market, having 5 of those on an
> ARM machine is really common and the ext4 limitation is becoming more
> reachable and requires attention.

It's not an ext4 limitation, though - it's a limitation of the pagecache.
With a 32-bit index into 4k pages, you can only address 16T in the
pagecache.  XFS won't mount it either, for example.

> What i'm trying to achieve is the following two items -
> 
> ---- item #1 ---
> Understand where the limitation is really coming from? Is this ext4
> implementation limitation or 32bit machines will never work with >16TB
> filesystems?

The latter, see above.

> I understand that there is a 16TB file size limitation (2^32*4K page
> size so you won't be able to mmap() further than that point) but how
> is that related to filesystem size?

fs metadata is mapped into an address space, IIRC, so can't be addressed
past 2^32 pages.  Also, mkfs can't do buffered IO to the device past
16T (it is writing to a device _file_) and ditto for e2fsck.

> Will 64KB page size fix this issue (ARM supports 4KB and 64KB pages) -
> clearly memory fragmentation will be a hit here.

If you can have 64k pages, I think you can address 2^32 * 64k.

> ----- item #2 ---
> Reproduce a failing scenario.
> For now i'v created a 24TB volume (thin provisioned) - RAID-0 on a 3 x
> loopback on a 3 x truncted 8TB consisting total of 24TB volume
> mkfs.ext4 /dev/md0 (e2fsprogrs 1.42 - thanks for the >16TB support)
> mount on a hacked kernel (#define pgoff_t unsigned long long thus
> making filesystem mounting check disappear)

that's the other way to do it; pgoff_t was made a typedef just
for that reason, but someone would need to audit a ton of code
to be sure it's used consistently, and doesn't overflow anywhere,
before it can be made larger.

Another thing to consider is whether you can successfully run e2fsck
on a very large filesystem on this box, even if you resolve the above
issues.  Would you have the resources you need to fsck, say, a 32T fs
if^Wwhen something goes wrong?

-Eric

> The volume mounts ok; now how do i get into corruption? I don't have
> physical 24TB drive, so best if there is a pin-pointed to test to
> reproduce the issue.
> 
> Best regards,
> Rabeeh
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 16 TB filesystem limit on 32bit machine

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2/20/2012 11:50 AM, Eric Sandeen wrote:
> fs metadata is mapped into an address space, IIRC, so can't be
> addressed past 2^32 pages.  Also, mkfs can't do buffered IO to the
> device past 16T (it is writing to a device _file_) and ditto for
> e2fsck.

But the file is only used for open(), after that the IO is handled by
the correct device driver, which handles 64 bit offsets ( when you
have CONFIG_LBDAF on ).  So if you can change the page cache index
size ( and it doesn't blow up ) this should work fine.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJPQrhrAAoJEJrBOlT6nu75yX8IAKNmJv0tH2CMWefUFV/4un0g
NKU5L2uwuMmyAzVRUOv165d1KAJkNc0z3OLUyh3425yku6wQ4dDqqnWSHCIZHJs7
/d1ltEP1g/vhpM5keAwwWHfC7N2KL+FRyqze8YAe6Qx5GGEWlUOqK2ALSxtuL2Il
TQ+lfLI9kAA5Ggtwf6mjjD/fKFTdJILQuZikxpJRfKasLg/4Y25LWdaPG8BOm78l
qPxkazpmIG8HVFczHwVWpuu+xY8nAmgDdZCTJBFoK9xmBUO0kjzl8JlAl4ScaXdi
v1793CSzev/2aIhEwXSK0HdI7e6KI4iT292/k4ryjfwJ+QX5L6eyfd4HlSVMM88=
=IOG0
-----END PGP SIGNATURE-----

Re: 16 TB filesystem limit on 32bit machine

[Eric Sandeen]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2/20/12 3:17 PM, Phillip Susi wrote:
> On 2/20/2012 11:50 AM, Eric Sandeen wrote:
>> fs metadata is mapped into an address space, IIRC, so can't be
>> addressed past 2^32 pages.  Also, mkfs can't do buffered IO to the
>> device past 16T (it is writing to a device _file_) and ditto for
>> e2fsck.
> 
> But the file is only used for open(), after that the IO is handled by
> the correct device driver, which handles 64 bit offsets ( when you
> have CONFIG_LBDAF on ).  So if you can change the page cache index
> size ( and it doesn't blow up ) this should work fine.

Oh, sure - if you change pgoff_t to 64 bits, but until then, you
can't even mkfs.ext4 a device larger than 16T; there is nowhere
for that buffered IO to go in the page cache, right?

(You could mkfs.ext4 a 16T partition on a 40T lun today, thanks
to LBDAF, but not a 17T without changing pgoff_t et al)

And "if you change the page cache index size" that's a big leap
of faith, today.  Might be interesting work to do, but I'd bet
it's a real chunk of work.

- -Eric
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJPQs5IAAoJECCuFpLhPd7gFGcP/REp9fxQJEXP/JoV8ixcdPmi
Grs15shrEuaJDTnz120wOVS+HSyYU7EczdC1uL6dHFtxqSm/gpdRka5zabz2L46l
RWcPjEofLjR45PWJaGzkWQrmrL9tyJdgP83idUUPl9AlGLbK4jgpzgs9OB0tf639
7lf/a711cZT1G7fLg02ZHb88TGE5BltEQHtX1nk1k4srLgjFKRCi5Am+auXgu1ta
W0Q3a+oxPlkOiVcr499xmInsAhPHBuErtd7B/S7ViP7Cz+Bhbv25xcM77jwRHmtp
9kwkt2ntQ4v9dccmlqpIMElqQJQGKU1li2ySzmJTUbS8jzmBXG/kXtUEr1y50tNc
tm6kIPkMX0RkXRIOfri2jq4LBV0Nl1uGIqUEbUvtJDMh9s4tBtlKV0ZWJi9foIaW
OqMAiEhvgb5tpMZG9gjfCSxnMelUAMC9LrygRG04O26Q9vQMEDD57Ee77MOTOZId
1g01fxZzcmd+UXlfDWmHtgSjotSMyp6pV0lWro82qe1pKo7HwwCjIA2KeHu9owTa
zmzvzbopd0OoUUmXzqAx2kBdxhhNqoqB2AQXfDf8tmyckA8YY4KMgJhKjFSfMc54
ZDzvWuJI11WoBsia9LlAbcFq0Dchd64Mq6As/By9N6QBWV0LdhkikSTFGTzKL4D2
FrAh6QhIiRhkaGiEI5XG
=2bla
-----END PGP SIGNATURE-----