Maximum file system size of XFS?

Maximum file system size of XFS?

[Pascal]

Hello,

I am asking you because I am insecure about the correct answer and
different sources give me different numbers.


My question is: What is the maximum file system size of XFS?

The official page says: 2^63 = 9 x 10^18 = 9 exabytes
Source: http://oss.sgi.com/projects/xfs/

Wikipedia says 16 exabytes.
Source: https://en.wikipedia.org/wiki/XFS

Another reference books says 8 exabytes (2^63).


Can anyone tell me and explain what is the maximum file system size for
XFS?


Thank you in advance!

Pascal

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Maximum file system size of XFS?

[Ric Wheeler]

On 03/09/2013 03:51 PM, Pascal wrote:
> Hello,
>
> I am asking you because I am insecure about the correct answer and
> different sources give me different numbers.
>
>
> My question is: What is the maximum file system size of XFS?
>
> The official page says: 2^63 = 9 x 10^18 = 9 exabytes
> Source: http://oss.sgi.com/projects/xfs/
>
> Wikipedia says 16 exabytes.
> Source: https://en.wikipedia.org/wiki/XFS
>
> Another reference books says 8 exabytes (2^63).
>
>
> Can anyone tell me and explain what is the maximum file system size for
> XFS?
>
>
> Thank you in advance!
>
> Pascal
>

The maximum size that XFS can address (which is what most people post in things 
like wikipedia) is kind of a fantasy number.

What is a better question is what is the maximum size XFS file system people 
have in production (even better, people who have your same work load). Lots and 
lots of tiny files are more challenging than very large video files for example.

I think that you can easily find people with 100's of terabytes in production 
use. For Red Hat, we support production use of 100TB per XFS instance in RHEL6 
for example since that is what we test at (and have been know to officially 
support larger instances by exception).

Some of the things to watch out for in very large file systems is how much DRAM 
you have in the server. If you ever need to xfs_repair a 1PB file system, you 
will need a very beefy box :)

Ric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Maximum file system size of XFS?

[Pascal]

Am Sat, 09 Mar 2013 17:29:23 -0500
schrieb Ric Wheeler <rwheeler@redhat.com>:

> On 03/09/2013 03:51 PM, Pascal wrote:
> > Hello,
> >
> > I am asking you because I am insecure about the correct answer and
> > different sources give me different numbers.
> >
> >
> > My question is: What is the maximum file system size of XFS?
> >
> > The official page says: 2^63 = 9 x 10^18 = 9 exabytes
> > Source: http://oss.sgi.com/projects/xfs/
> >
> > Wikipedia says 16 exabytes.
> > Source: https://en.wikipedia.org/wiki/XFS
> >
> > Another reference books says 8 exabytes (2^63).
> >
> >
> > Can anyone tell me and explain what is the maximum file system size
> > for XFS?
> >
> >
> > Thank you in advance!
> >
> > Pascal
> >
> 
> The maximum size that XFS can address (which is what most people post
> in things like wikipedia) is kind of a fantasy number.
> 
> What is a better question is what is the maximum size XFS file system
> people have in production (even better, people who have your same
> work load). Lots and lots of tiny files are more challenging than
> very large video files for example.
> 
> I think that you can easily find people with 100's of terabytes in
> production use. For Red Hat, we support production use of 100TB per
> XFS instance in RHEL6 for example since that is what we test at (and
> have been know to officially support larger instances by exception).
> 
> Some of the things to watch out for in very large file systems is how
> much DRAM you have in the server. If you ever need to xfs_repair a
> 1PB file system, you will need a very beefy box :)
> 
> Ric
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
> 

Hello Ric,

thank you for your answer. I am aware that there is a difference
between the maximum size under practical conditions and the theoretical
maximum. But I am looking for this theoretical number to use in within
in my thesis comparing file systems.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Maximum file system size of XFS?

[Eric Sandeen]

On 3/9/13 4:39 PM, Pascal wrote:

> Hello Ric,
> 
> thank you for your answer. I am aware that there is a difference
> between the maximum size under practical conditions and the theoretical
> maximum. But I am looking for this theoretical number to use in within
> in my thesis comparing file systems.

A thesis comparing actual scalability would be much more interesting
than one comparing, essentially, the container size chosen for a disk
block.  One could quickly write a filesystem which "can" be as large
as a yottabyte, but it wouldn't really *mean* anything.

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Maximum file system size of XFS?

[Stan Hoeppner]

On 3/9/2013 7:10 PM, Eric Sandeen wrote:
> On 3/9/13 4:39 PM, Pascal wrote:
> 
>> Hello Ric,
>>
>> thank you for your answer. I am aware that there is a difference
>> between the maximum size under practical conditions and the theoretical
>> maximum. But I am looking for this theoretical number to use in within
>> in my thesis comparing file systems.
> 
> A thesis comparing actual scalability would be much more interesting
> than one comparing, essentially, the container size chosen for a disk
> block.  One could quickly write a filesystem which "can" be as large
> as a yottabyte, but it wouldn't really *mean* anything.

Agreed.  But if the OP must have the theoretical maximum, I think what's
in the SGI doc is the correct number.  Down below what the OP quoted
from the Features section, down in the Technical Specifications, we find:

" Maximum Filesystem Size

For Linux 2.4, 2 TB. For Linux 2.6 and beyond, when using 64 bit
addressing in the block devices layer (CONFIG_LBD) and a 64 bit
platform, filesystem size limit increases to 9 million terabytes (or the
device limits). For these later kernels on 32 bit platforms, 16TB is the
current limit even with 64 bit addressing enabled in the block layer."


I assume the OP's paper deals with the far distant future where
individual rusty disk drives have 1PB capacity, thus requiring 'only'
9,000 disk drives for a RAW 9EB XFS without redundancy, or 18,000 drives
for RAID10.  With today's largest drives at 4TB, it would take 2.25
million disk drives for a RAW 9EB capacity, 4.5 million for RAID10.  All
of this assuming my math is correct.  I don't regularly deal with 16
digit decimal numbers. ;)  I'm also assuming in this distant future that
rusty drives still lead SSD in price/capacity.  That may be an incorrect
assumption.  Dave can beat up on me in a couple of decades if my
assumption proves incorrect. ;)

For a 9EB XFS to become remotely practical, I'd say disk drive capacity
would have to reach 10 petabytes per drive.  This yields 1800 drives for
9EB in RAID10, or 3x 42U racks each housing 10x 4U 60 drive FC RAID
chassis, 600 drives per rack.  I keep saying RAID10 instead of RAID6
because I don't think anyone would want to attempt a RAID6 parity
rebuild of even a small 4+2 array of 10PB drives, if the sustained
interface rate continues to increase at the snails pace it has in
relation to aerial density.  Peak interface sustained data rate today is
about 200MB/s for the fastest rusty drives.  If we are lucky the 10PB
drives of the future will have a sustained interface rate of 20GB/s, or
100x today's fastest, which will allow for a mirroring operation to
complete in about 14 hours, which is still slower than with today's 4TB
drives, which take about 8 hours.

Note that a 20GB/s one way data rate of such a 10PB drive would saturate
a 16 lane PCI Express v3.0 slot (15GB/s), and eat 2/3rds of a v4.0 x16
slot's bandwidth (31GB/s, but won't ship until ~2016).  And since
current PCIe controller to processor interconnects are limited to about
12-20GB/s one way, PCIe b/w doesn't matter.  Thus, the throughput of our
our peripheral and system level interconnects much increase many fold as
well to facilitate the hardware that would enable an EB sized XFS.

And as Ric mentioned, the memory capacity requirements for executing
xfs_repair on a 9EB XFS would likely require a host machine with many
hundreds of times the memory capacity of system available today.  That
and/or a rewrite of xfs_repair to make more efficient use of RAM.

So in summary, an Exabyte scale XFS is simply not practical today, and
won't be for at least another couple of decades, or more, if ever.  The
same holds true for some of the other filesystems you're going to be
writing about.  Some of the cluster and/or distributed filesystems
you're looking at could probably scale to Exabytes today.  That is, if
someone had the budget for half a million hard drives, host systems,
switches, etc, the facilities to house it all, and the budget for power
and cooling.  That's 834 racks for drives alone, just under 1/3rd of a
mile long if installed in a single row.

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Maximum file system size of XFS?

[Stan Hoeppner]

On 3/10/2013 1:54 AM, Stan Hoeppner wrote:

> So in summary, an Exabyte scale XFS is simply not practical today, and
> won't be for at least another couple of decades, or more, if ever.  The
> same holds true for some of the other filesystems you're going to be
> writing about.  Some of the cluster and/or distributed filesystems
> you're looking at could probably scale to Exabytes today.  That is, if
> someone had the budget for half a million hard drives, host systems,
> switches, etc, the facilities to house it all, and the budget for power
> and cooling.  That's 834 racks for drives alone, just under 1/3rd of a
> mile long if installed in a single row.

Jet lag due to time travel caused a math error above.  With today's 4TB
drives it would require 2.25 million units for a raw 9EB capacity.
That's 3,750 racks of 600 drives each.  These would stretch 1.42 miles,
7500 ft.

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Maximum file system size of XFS?

[Hans-Peter Jansen]

Am Montag, 11. März 2013, 06:02:26 schrieb Stan Hoeppner:
> On 3/10/2013 1:54 AM, Stan Hoeppner wrote:
> > So in summary, an Exabyte scale XFS is simply not practical today, and
> > won't be for at least another couple of decades, or more, if ever.  The
> > same holds true for some of the other filesystems you're going to be
> > writing about.  Some of the cluster and/or distributed filesystems
> > you're looking at could probably scale to Exabytes today.  That is, if
> > someone had the budget for half a million hard drives, host systems,
> > switches, etc, the facilities to house it all, and the budget for power
> > and cooling.  That's 834 racks for drives alone, just under 1/3rd of a
> > mile long if installed in a single row.
> 
> Jet lag due to time travel caused a math error above.  With today's 4TB
> drives it would require 2.25 million units for a raw 9EB capacity.
> That's 3,750 racks of 600 drives each.  These would stretch 1.42 miles,
> 7500 ft.

And I just acknowledged the building plans for our new datacenter, based on 
your former calculations. The question is, who carries the costs of the now 
needed 4 other floors of that building.. 

Are you well-insured, Stan?

Cheers,
Pete

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Maximum file system size of XFS?

[Emmanuel Florac]

Le Mon, 11 Mar 2013 17:15:08 +0100
Hans-Peter Jansen <hpj@urpla.net> écrivait:

> And I just acknowledged the building plans for our new datacenter,
> based on your former calculations.

Don't be afraid, there are 80 drives 4U chassis available, and 5TB
drives are around the corner. That's 800 drives and a raw capacity of
4 PB per 42U rack.

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Maximum file system size of XFS?

[Dave Chinner]

On Sat, Mar 09, 2013 at 11:39:40PM +0100, Pascal wrote:
> thank you for your answer. I am aware that there is a difference
> between the maximum size under practical conditions and the theoretical
> maximum. But I am looking for this theoretical number to use in within
> in my thesis comparing file systems.

Out of curiousity, what apsect of file systems are you comparing?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Maximum file system size of XFS?

[Martin Steigerwald]

Am Samstag, 9. März 2013 schrieb Pascal:
> Hello,

Hi Pascal,

> I am asking you because I am insecure about the correct answer and
> different sources give me different numbers.
> 
> 
> My question is: What is the maximum file system size of XFS?
> 
> The official page says: 2^63 = 9 x 10^18 = 9 exabytes
> Source: http://oss.sgi.com/projects/xfs/
> 
> Wikipedia says 16 exabytes.
> Source: https://en.wikipedia.org/wiki/XFS
> 
> Another reference books says 8 exabytes (2^63).
> 
> 
> Can anyone tell me and explain what is the maximum file system size for
> XFS?

You can test it. The theoretical limit. Whether such a filesystem will work 
nicely with a real workload is, as pointed out, a different question.

1) Use a big enough XFS filesystem (yes, it has to be XFS for anything else 
that can carry a exabyte big sparse file)

merkaba:~> LANG=C mkfs.xfs -L justcrazy /dev/merkaba/zeit
meta-data=/dev/merkaba/zeit      isize=256    agcount=4, agsize=1310720 blks
         =                       sectsz=512   attr=2, projid32bit=0
data     =                       bsize=4096   blocks=5242880, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0


2) Create a insanely big sparse file

merkaba:~> truncate -s1E /mnt/zeit/evenmorecrazy.img
merkaba:~> ls -lh /mnt/zeit/evenmorecrazy.img
-rw-r--r-- 1 root root 1,0E Mär 11 22:37 /mnt/zeit/evenmorecrazy.img

(No, this won´t work with Ext4.)


3) Make a XFS file system into it:

merkaba:~> mkfs.xfs -L /mnt/zeit/evenmorecrazy.img

I won´t today. I tried that for gag during a linux performance and analysis 
training I held on a ThinkPad T520 with Sandybridge i5 2,50 GhZ, Intel SSD 
320 on an about 20 GiB XFS filesystem.

The mkfs command run for something like one or two hours. It was using quite 
some CPU and quite some SSD, but did not max out one of it.

The host XFS filesystem was almost full, so the image took just about those 
20 GiB.


4) Mount it and enjoy the output of df -hT.


5) Write to if it you dare. I did it, until the Linux kernel told something 
about "lost buffer writes". What I found strange is, that the dd writing to 
the 1E filesystem did not quit then with input/output error. It just ran on.


I didn´t test this with any larger size, but if size and time usage scales 
linearily it might be possible to create a 10EiB filesystem within 200 GiB 
host XFS and hum about a day of waiting :).

No, I do not suggest to use anything even just remotely like this in 
production.

And no, my test didn´t show that an 1EiB filesystem will work nicely with 
any real life workload.

Am I crazy for trying this? I might be :)

Thanks,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Maximum file system size of XFS?

[Martin Steigerwald]

Am Montag, 11. März 2013 schrieb Martin Steigerwald:
> 2) Create a insanely big sparse file
> 
> merkaba:~> truncate -s1E /mnt/zeit/evenmorecrazy.img
> merkaba:~> ls -lh /mnt/zeit/evenmorecrazy.img
> -rw-r--r-- 1 root root 1,0E Mär 11 22:37 /mnt/zeit/evenmorecrazy.img
> 
> (No, this won´t work with Ext4.)

Okay, you can´t go beyond 8 EiB for a single file which is about what I have 
read somewhere:

merkaba:/mnt/zeit> ls -lh
insgesamt 0
-rw-r--r-- 1 root root 1,0E Mär 11 22:37 evenmorecrazy.img
merkaba:/mnt/zeit> truncate -s2E /mnt/zeit/evenmorecrazy.img
merkaba:/mnt/zeit> truncate -s3E /mnt/zeit/evenmorecrazy.img
merkaba:/mnt/zeit> truncate -s4E /mnt/zeit/evenmorecrazy.img
merkaba:/mnt/zeit> truncate -s5E /mnt/zeit/evenmorecrazy.img
merkaba:/mnt/zeit> truncate -s6E /mnt/zeit/evenmorecrazy.img
merkaba:/mnt/zeit> truncate -s7E /mnt/zeit/evenmorecrazy.img
merkaba:/mnt/zeit> LANG=C truncate -s8E /mnt/zeit/evenmorecrazy.img
truncate: invalid number '8E': Value too large for defined data type
merkaba:/mnt/zeit#1> ls -lh  
insgesamt 0
-rw-r--r-- 1 root root 7,0E Mär 11 22:49 evenmorecrazy.img

So so tests stops there, until you concatenate two of those files with LVM 
or SoftRAID 0 (if that works). Like this (I just had to try it):


merkaba:/mnt/zeit> ls -lh                                           
insgesamt 0
-rw-r--r-- 1 root root 7,0E Mär 11 22:49 evenmorecrazy.img
-rw-r--r-- 1 root root 7,0E Mär 11 22:52 evenmorecrazy.img2
merkaba:/mnt/zeit> losetup /dev/loop0 evenmorecrazy.img
merkaba:/mnt/zeit> losetup /dev/loop1 evenmorecrazy.img2

merkaba:/mnt/zeit#5> pvcreate /dev/loop0       
  Physical volume "/dev/loop0" successfully created
merkaba:/mnt/zeit> pvcreate /dev/loop1
  Physical volume "/dev/loop1" successfully created
merkaba:/mnt/zeit> vgcreate justinsane /dev/loop0 /dev/loop1
  PV /dev/loop0 too large for extent size 4,00 MiB.
  Format-specific setup of physical volume '/dev/loop0' failed.
  Unable to add physical volume '/dev/loop0' to volume group 'justinsane'.


merkaba:/mnt/zeit#5> vgcreate --physicalextentsize 16M  justinsane 
/dev/loop0 /dev/loop1
  PV /dev/loop0 too large for extent size 16,00 MiB.
  Format-specific setup of physical volume '/dev/loop0' failed.
  Unable to add physical volume '/dev/loop0' to volume group 'justinsane'.

merkaba:/mnt/zeit#5> vgcreate --physicalextentsize 128M  justinsane 
/dev/loop0 /dev/loop1
  PV /dev/loop0 too large for extent size 128,00 MiB.
  Format-specific setup of physical volume '/dev/loop0' failed.
  Unable to add physical volume '/dev/loop0' to volume group 'justinsane'.

merkaba:/mnt/zeit#5> vgcreate --physicalextentsize 1G  justinsane /dev/loop0 
/dev/loop1
  PV /dev/loop0 too large for extent size 1,00 GiB.
  Format-specific setup of physical volume '/dev/loop0' failed.
  Unable to add physical volume '/dev/loop0' to volume group 'justinsane'.

merkaba:/mnt/zeit#5> vgcreate --physicalextentsize 4G  justinsane /dev/loop0 
/dev/loop1
  Volume group "justinsane" successfully created

merkaba:/mnt/zeit> vgs                                                               
  VG         #PV #LV #SN Attr   VSize   VFree 
  justinsane   2   0   0 wz--n-  14,00e 14,00e
  merkaba      1   4   0 wz--n- 278,99g  4,85g
merkaba:/mnt/zeit>

merkaba:/mnt/zeit> vgdisplay justinsane
  --- Volume group ---
  VG Name               justinsane
  System ID             
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  1
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                0
  Open LV               0
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               14,00 EiB
  PE Size               4,00 GiB
  Total PE              3758096382
  Alloc PE / Size       0 / 0   
  Free  PE / Size       3758096382 / 14,00 EiB
  VG UUID               z8JP5s-lfRw-uKo8-DXAP-XWGe-aKra-xug9Nn


Enough insanity for today :)

I won´t mkfs.xfs on it, the 20 GiB of the just filesystem wouldn´t be 
enough.

Thanks,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Maximum file system size of XFS?

[Martin Steigerwald]

Am Montag, 11. März 2013 schrieb Martin Steigerwald:
> Am Montag, 11. März 2013 schrieb Martin Steigerwald:
> > 2) Create a insanely big sparse file
> > 
> > merkaba:~> truncate -s1E /mnt/zeit/evenmorecrazy.img
> > merkaba:~> ls -lh /mnt/zeit/evenmorecrazy.img
> > -rw-r--r-- 1 root root 1,0E Mär 11 22:37 /mnt/zeit/evenmorecrazy.img
> > 
> > (No, this won´t work with Ext4.)
> 
> Okay, you can´t go beyond 8 EiB for a single file which is about what I
> have read somewhere:
[…]
> merkaba:/mnt/zeit> truncate -s7E /mnt/zeit/evenmorecrazy.img
> merkaba:/mnt/zeit> LANG=C truncate -s8E /mnt/zeit/evenmorecrazy.img
> truncate: invalid number '8E': Value too large for defined data type
> merkaba:/mnt/zeit#1> ls -lh
> insgesamt 0
> -rw-r--r-- 1 root root 7,0E Mär 11 22:49 evenmorecrazy.img
> 
> So so tests stops there, until you concatenate two of those files with
> LVM or SoftRAID 0 (if that works). Like this (I just had to try it):
> 
> 
> merkaba:/mnt/zeit> ls -lh
> insgesamt 0
> -rw-r--r-- 1 root root 7,0E Mär 11 22:49 evenmorecrazy.img
> -rw-r--r-- 1 root root 7,0E Mär 11 22:52 evenmorecrazy.img2
> merkaba:/mnt/zeit> losetup /dev/loop0 evenmorecrazy.img
> merkaba:/mnt/zeit> losetup /dev/loop1 evenmorecrazy.img2
> 
> merkaba:/mnt/zeit#5> pvcreate /dev/loop0
>   Physical volume "/dev/loop0" successfully created
> merkaba:/mnt/zeit> pvcreate /dev/loop1
>   Physical volume "/dev/loop1" successfully created
> merkaba:/mnt/zeit> vgcreate justinsane /dev/loop0 /dev/loop1
>   PV /dev/loop0 too large for extent size 4,00 MiB.
>   Format-specific setup of physical volume '/dev/loop0' failed.
>   Unable to add physical volume '/dev/loop0' to volume group
> 'justinsane'.
[…]
> merkaba:/mnt/zeit#5> vgcreate --physicalextentsize 4G  justinsane
> /dev/loop0 /dev/loop1
>   Volume group "justinsane" successfully created
> 
> merkaba:/mnt/zeit> vgs
>   VG         #PV #LV #SN Attr   VSize   VFree
>   justinsane   2   0   0 wz--n-  14,00e 14,00e
>   merkaba      1   4   0 wz--n- 278,99g  4,85g
> merkaba:/mnt/zeit>
> 
> merkaba:/mnt/zeit> vgdisplay justinsane
[…]
> Enough insanity for today :)

Not quite:

> I won´t mkfs.xfs on it, the 20 GiB of the just filesystem wouldn´t be
> enough.

Ok, there seems to be another limit involved:

merkaba:/mnt/zeit> lvcreate -n yourbiggiexfs -L14E justinsane
  Volume group "justinsane" has insufficient free space (3758096382 
extents): 3758096384 required.
merkaba:/mnt/zeit#5> LANG=C lvcreate -n yourbiggiexfs -L14E justinsane
  Volume group "justinsane" has insufficient free space (3758096382 
extents): 3758096384 required.
merkaba:/mnt/zeit#5> LANG=C lvcreate -n yourbiggiexfs -L13E justinsane
  /dev/justinsane/yourbiggiexfs: lseek 0 failed: Invalid argument
  /dev/justinsane/yourbiggiexfs: lseek 0 failed: Invalid argument
  Logical volume "yourbiggiexfs" created
merkaba:/mnt/zeit> LANG=C lvs                                       
  /dev/justinsane/yourbiggiexfs: lseek 14987979559888945152 failed: Invalid 
argument
  /dev/justinsane/yourbiggiexfs: lseek 14987979559889002496 failed: Invalid 
argument
  /dev/justinsane/yourbiggiexfs: lseek 0 failed: Invalid argument
  /dev/justinsane/yourbiggiexfs: lseek 4096 failed: Invalid argument
  /dev/justinsane/yourbiggiexfs: lseek 0 failed: Invalid argument
  LV            VG         Attr      LSize   Pool Origin Data%  Move Log 
Copy%  Convert
  yourbiggiexfs justinsane -wi-a----  13.00e 

So testing with 9 EiB might become some issue :)

Thanks,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Maximum file system size of XFS?

[Martin Steigerwald]

Am Montag, 11. März 2013 schrieb Martin Steigerwald:
> Am Montag, 11. März 2013 schrieb Martin Steigerwald:
> > Am Montag, 11. März 2013 schrieb Martin Steigerwald:
> > > 2) Create a insanely big sparse file
> > > 
> > > merkaba:~> truncate -s1E /mnt/zeit/evenmorecrazy.img
> > > merkaba:~> ls -lh /mnt/zeit/evenmorecrazy.img
> > > -rw-r--r-- 1 root root 1,0E Mär 11 22:37 /mnt/zeit/evenmorecrazy.img
> > > 
> > > (No, this won´t work with Ext4.)
> > 
> > Okay, you can´t go beyond 8 EiB for a single file which is about what I
> 
> > have read somewhere:
> […]
> 
> > merkaba:/mnt/zeit> truncate -s7E /mnt/zeit/evenmorecrazy.img
> > merkaba:/mnt/zeit> LANG=C truncate -s8E /mnt/zeit/evenmorecrazy.img
> > truncate: invalid number '8E': Value too large for defined data type
> > merkaba:/mnt/zeit#1> ls -lh
> > insgesamt 0
> > -rw-r--r-- 1 root root 7,0E Mär 11 22:49 evenmorecrazy.img
> > 
> > So so tests stops there, until you concatenate two of those files with
> > LVM or SoftRAID 0 (if that works). Like this (I just had to try it):
> > 
> > 
> > merkaba:/mnt/zeit> ls -lh
> > insgesamt 0
> > -rw-r--r-- 1 root root 7,0E Mär 11 22:49 evenmorecrazy.img
> > -rw-r--r-- 1 root root 7,0E Mär 11 22:52 evenmorecrazy.img2
> > merkaba:/mnt/zeit> losetup /dev/loop0 evenmorecrazy.img
> > merkaba:/mnt/zeit> losetup /dev/loop1 evenmorecrazy.img2
> > 
> > merkaba:/mnt/zeit#5> pvcreate /dev/loop0
> > 
> >   Physical volume "/dev/loop0" successfully created
> > 
> > merkaba:/mnt/zeit> pvcreate /dev/loop1
> > 
> >   Physical volume "/dev/loop1" successfully created
> > 
> > merkaba:/mnt/zeit> vgcreate justinsane /dev/loop0 /dev/loop1
> > 
> >   PV /dev/loop0 too large for extent size 4,00 MiB.
> >   Format-specific setup of physical volume '/dev/loop0' failed.
> >   Unable to add physical volume '/dev/loop0' to volume group
> > 
> > 'justinsane'.
> 
> […]
> 
> > merkaba:/mnt/zeit#5> vgcreate --physicalextentsize 4G  justinsane
> > /dev/loop0 /dev/loop1
> > 
> >   Volume group "justinsane" successfully created
> > 
> > merkaba:/mnt/zeit> vgs
> > 
> >   VG         #PV #LV #SN Attr   VSize   VFree
> >   justinsane   2   0   0 wz--n-  14,00e 14,00e
> >   merkaba      1   4   0 wz--n- 278,99g  4,85g
> > 
> > merkaba:/mnt/zeit>
> > 
> > merkaba:/mnt/zeit> vgdisplay justinsane
> 
> […]
> 
> > Enough insanity for today :)
> 
> Not quite:
> > I won´t mkfs.xfs on it, the 20 GiB of the just filesystem wouldn´t be
> > enough.
> 
> Ok, there seems to be another limit involved:
> 
> merkaba:/mnt/zeit> lvcreate -n yourbiggiexfs -L14E justinsane
>   Volume group "justinsane" has insufficient free space (3758096382
> extents): 3758096384 required.
[…]
> merkaba:/mnt/zeit#5> LANG=C lvcreate -n yourbiggiexfs -L13E justinsane
>   /dev/justinsane/yourbiggiexfs: lseek 0 failed: Invalid argument
>   /dev/justinsane/yourbiggiexfs: lseek 0 failed: Invalid argument
>   Logical volume "yourbiggiexfs" created
> merkaba:/mnt/zeit> LANG=C lvs
>   /dev/justinsane/yourbiggiexfs: lseek 14987979559888945152 failed:
> Invalid argument
>   /dev/justinsane/yourbiggiexfs: lseek 14987979559889002496 failed:
> Invalid argument

Well,

merkaba:/mnt/zeit> LANG=C ls -l
total 24
-rw-r--r-- 1 root root 8070450532247928832 Mar 11 23:02 evenmorecrazy.img
-rw-r--r-- 1 root root 8070450532247928832 Mar 11 23:02 evenmorecrazy.img2

looks crazy enough for me already.

(Ok, I really stop this now :)

Thanks,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Maximum file system size of XFS?

[Pascal]

Am Mon, 11 Mar 2013 23:04:14 +0100
schrieb Martin Steigerwald <Martin@lichtvoll.de>:

> > > Enough insanity for today :)

Hey Martin,

I am very impressed by the time and effort you spent into it! Thank you!


And thanks to everyone for answering my question!



_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Maximum file system size of XFS?

[Dave Chinner]

On Mon, Mar 11, 2013 at 10:57:00PM +0100, Martin Steigerwald wrote:
> Am Montag, 11. März 2013 schrieb Martin Steigerwald:
> > 2) Create a insanely big sparse file
> > 
> > merkaba:~> truncate -s1E /mnt/zeit/evenmorecrazy.img
> > merkaba:~> ls -lh /mnt/zeit/evenmorecrazy.img
> > -rw-r--r-- 1 root root 1,0E Mär 11 22:37 /mnt/zeit/evenmorecrazy.img
> > 
> > (No, this won´t work with Ext4.)
> 
> Okay, you can´t go beyond 8 EiB for a single file which is about what I have 
> read somewhere:

Right - file size offsets max out at 2^63 bytes.

....

> merkaba:/mnt/zeit#5> vgcreate --physicalextentsize 4G  justinsane /dev/loop0 
> /dev/loop1
>   Volume group "justinsane" successfully created
> 
> merkaba:/mnt/zeit> vgs                                                               
>   VG         #PV #LV #SN Attr   VSize   VFree 
>   justinsane   2   0   0 wz--n-  14,00e 14,00e
>   merkaba      1   4   0 wz--n- 278,99g  4,85g
> merkaba:/mnt/zeit>
....
> Enough insanity for today :)
> 
> I won´t mkfs.xfs on it, the 20 GiB of the just filesystem wouldn´t be 
> enough.

Right - I did a mkfs.xfs on a (8EB - 1GB) file a couple of days ago
just to check it worked. I killed it after a short while, because I
didn't feel like needlessly subjecting the SSDs the file was
physically located on to the 25 million sparse sector sized writes
needed for mkfs to complete.

And you can double that number of writes needed for a 16EB
filesystem to be initialised by mkfs. So, theory be damned, even
mkfs.xfs doesn't scale to supporting exabyte filesystems...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Maximum file system size of XFS?

[Martin Steigerwald]

Am Montag, 11. März 2013 schrieb Martin Steigerwald:
> Am Samstag, 9. März 2013 schrieb Pascal:
> > Hello,
> 
> Hi Pascal,
> 
> > I am asking you because I am insecure about the correct answer and
> > different sources give me different numbers.
> >
> > 
> > 
> >
> > My question is: What is the maximum file system size of XFS?
> >
> > 
> >
> > The official page says: 2^63 = 9 x 10^18 = 9 exabytes
> > Source: http://oss.sgi.com/projects/xfs/
> >
> > 
> >
> > Wikipedia says 16 exabytes.
> > Source: https://en.wikipedia.org/wiki/XFS
> >
> > 
> >
> > Another reference books says 8 exabytes (2^63).
> >
> > 
> > 
> >
> > Can anyone tell me and explain what is the maximum file system size for
> > XFS?
> 
> You can test it. The theoretical limit. Whether such a filesystem will
> work  nicely with a real workload is, as pointed out, a different
> question.

Well, as I just demonstrated, you can´t. At least not with XFS within XFS. 
You can only test for maximum filesize. If XFS can be larger than that, you 
need a filesystem which can carry an even larger file. Maybe ZFS on Linux or 
so?

BTRFS doesn´t go beyond 8 EiB per file as well:

merkaba:/mnt#1> mkfs.btrfs -n 16384 -l 16384 /dev/merkaba/zeit

WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using

fs created label (null) on /dev/merkaba/zeit
        nodesize 16384 leafsize 16384 sectorsize 4096 size 20.00GB
Btrfs Btrfs v0.19
merkaba:/mnt> mount /dev/merkaba/zeit zeit
merkaba:/mnt> cd zeit
merkaba:/mnt/zeit> truncate -s1E canitgetcrazierthanthat.img
merkaba:/mnt/zeit> truncate -s2E canitgetcrazierthanthat.img
merkaba:/mnt/zeit> truncate -s3E canitgetcrazierthanthat.img
merkaba:/mnt/zeit> truncate -s4E canitgetcrazierthanthat.img
merkaba:/mnt/zeit> truncate -s5E canitgetcrazierthanthat.img
merkaba:/mnt/zeit> truncate -s6E canitgetcrazierthanthat.img
merkaba:/mnt/zeit> truncate -s7E canitgetcrazierthanthat.img
merkaba:/mnt/zeit> truncate -s8E canitgetcrazierthanthat.img
truncate: ungültige Zahl „8E“: Der Wert ist zu groß für den definierten 
Datentyp
merkaba:/mnt/zeit#1> LANG=C truncate -s8E canitgetcrazierthanthat.im
truncate: invalid number '8E': Value too large for defined data type

merkaba:/mnt/zeit#1> ls -lh
insgesamt 0
-rw-r--r-- 1 root root 7,0E Mär 11 23:09 canitgetcrazierthanthat.img

Thanks,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs